How the Intelligence Score works智能评分是怎么算的

Raw benchmark numbers are not comparable — 90 on MMLU-Pro is routine, 30 on FrontierMath is world-class. So every benchmark is normalized against the best result among the models we track: the leader sets index 100, everyone else is a fraction of it. LMArena Elo uses a difference scale instead (400 Elo behind the leader ≈ index 0, the standard chess interpretation). Anchors re-baseline automatically with every data update — when a new model raises the bar, the whole table re-anchors. Benchmarks do saturate: MMLU-Pro and AIME are on our watchlist and will be replaced or down-weighted when they stop discriminating.

原始分数之间没法直接比——MMLU-Pro 考 90 很平常,FrontierMath 考 30 就是世界级。所以每个 benchmark 都按"被追踪模型里的最好成绩"归一:榜首 = 指数 100,其他人按占比折算。LMArena Elo 改用分差制(落后榜首 400 分 ≈ 指数 0,国际象棋标准解释)。锚点随每次数据更新自动重定——新模型抬高了天花板,整张榜就重新锚定。Benchmark 会饱和——MMLU-Pro 和 AIME 已在观察名单,失去区分度时会降权或替换。

The final score is the weighted average over the benchmarks a model actually has, renormalized by the weights present. Coverage = share of total weight with data. Below 30% coverage we publish no score at all; below 60% the model carries a "low data" badge. We do this because silent renormalization is how leaderboards quietly lie — a model with two easy benchmarks should never outrank one measured on eleven.

总分 = 模型实际拥有的 benchmark 按权重加权平均,按出席权重重归一。覆盖率 = 有数据的权重占比。低于 30% 不出分;低于 60% 打"数据少"徽标。为什么?因为排行榜最常见的撒谎方式就是悄悄重归一——只考过两门简单科目的模型,不该排在考完全部十一门的模型前面。

For each (model, benchmark) pair we keep exactly one score, chosen by trust: official leaderboard first, then established third-party measurers (Epoch AI, vals.ai, etc.), and only then vendor self-reported numbers. Models whose data is >40% vendor self-reported carry a "vendor data" badge. LMArena stays in the mix at 5% weight — it is the only large-scale human-preference signal, but after 2025's "Leaderboard Illusion" findings we cap its influence. GAIA is excluded entirely after the 2026 exploit scandal; WebArena and τ²-bench entries note the agent scaffold used, since agent scores are setup-sensitive.

每个(模型 × benchmark)只保留一条成绩,按信任度取舍:官方榜优先,其次是老牌第三方(Epoch AI、vals.ai 等),最后才是厂商自报。厂商自报占比超 40% 的模型会打"厂商数据"徽标。LMArena 保留 5% 权重——它是唯一大规模真人偏好信号,但 2025 年《排行榜幻觉》事件后我们压低它的影响。GAIA 因 2026 年刷榜丑闻整体剔除;WebArena 和 τ²-bench 的成绩标注所用脚手架,因为 Agent 分数对测试方式高度敏感。

When several sources report a score for the same (model, benchmark), we keep all of them in the raw dataset and show a ×N badge beside the score. If sources disagree by more than 3 points (30 Elo for LMArena), the score carries an ⚠± flag — disagreement usually means different prompt setups, agent scaffolds, or vendor optimism (self-reported numbers are compared against independent measurements, not hidden). Scoring always uses the single most trustworthy entry; the flag is transparency, not a penalty. Scores are re-collected by a rotating daily cron: LMArena every day, each benchmark at least weekly, from official leaderboards and established aggregators.

同一(模型 × benchmark)有多个来源报分时,原始数据全部保留,成绩旁标 ×N 徽标。多源分差超过 3 分(LMArena 为 30 Elo)时打 ⚠± 标记——分差通常来自提示词设置、Agent 脚手架或厂商乐观口径(厂商自报会与第三方实测并列对照,而非隐藏)。算分始终只用可信度最高的一条;标记是透明化提示,不影响分数。成绩由每日轮换 cron 重新采集:LMArena 每日一次,每个 benchmark 至少每周一轮,来源为官方榜单与老牌第三方聚合站。

Many frontier models can spend more compute at inference time — OpenAI minimal/low/medium/high/xhigh, Gemini low/high, GLM and Grok effort levels, thinking switches on Qwen/DeepSeek/Kimi. The same model can differ by double digits between tiers, so a tier-blind comparison is apples-to-oranges. We record the tier every score was measured at: a ⚡ badge sits beside the score (⚡off = non-thinking mode), and when a model was measured at several tiers, the model page lists each tier as its own indented sub-row. The Intelligence Score uses the highest-tier trustworthy entry per benchmark — the capability ceiling, same convention as public leaderboards. Spread and ⚠± flags are computed within the same tier, so a legitimate cross-tier gap never reads as a source disagreement. In the rankings table, each model name carries its top supported tier as a formal suffix — GLM-5.2 (max), Qwen3.7-Max (on/off), Kimi K3 (always-on) — a vendor-documented capability curated by hand; no suffix means we found no official documentation, not that the model lacks tiers.

许多前沿模型在推理时可以花更多算力——OpenAI 的 minimal/low/medium/high/xhigh、Gemini 的 low/high、GLM 与 Grok 的 effort 档、Qwen/DeepSeek/Kimi 的 thinking 开关。同一模型不同档位能差出两位数,不看档位的对比等于苹果比橘子。我们记录每个分数的实测档位:成绩旁有 ⚡ 徽章(⚡off=非推理模式);一个模型测过多档时,模型页会把每一档列为缩进子行逐档对比。智能评分在每个 benchmark 上采用可信度最高的高档位成绩——即能力上限,与公开榜单惯例一致。分差与 ⚠± 标记只在同档位内计算,合理的跨档差距不会被误报成"来源打架"。排行榜里每个模型的名字后面带它支持的最高档作为正式后缀——GLM-5.2 (max)、Qwen3.7-Max (开/关)、Kimi K3 (常开)——这是厂商文档记载的能力规格,人工策展;名字没有后缀表示未查到官方文档,不代表模型没有档位。

Model catalog, context windows and API prices: official OpenRouter API (public, no key). Human preference: LMArena leaderboard data, CC-BY-4.0 (Arena AI, Inc.). Benchmark scores: the maintainers listed on each benchmark card, cited per entry on every model page. We link sources instead of mirroring datasets where licenses restrict commercial reuse. Rankings refresh with each data update; pricing is re-synced from OpenRouter on a daily cron.

模型目录、上下文与 API 价格:OpenRouter 官方公开 API(无需密钥)。人类偏好:LMArena 榜单数据,CC-BY-4.0(Arena AI, Inc.)。Benchmark 成绩:见各 benchmark 卡片列出的维护方,模型页逐条附来源链接。许可限制商用转载的数据集,我们只链接不镜像。榜单随数据更新重建;价格由每日 cron 从 OpenRouter 同步。