怎么按官方口径读一个模型的评测说明How to read a model's evaluation write-up on its official terms从业者视角的方法论 · 关键事实挂官方出处practitioner-perspective methodology · key facts carry official sources

这一篇讲的是怎么读别人写的评测说明，不是怎么替它下结论。

评测说明是一份条件句：谁在什么任务集上、用什么指标、和哪个版本比。四个成分齐全才算读到位。

本站不跑评测、不复现分数，也不在任何档位之间做优劣排名。家族卡上的能力声明一律逐条挂出处，出处之外留空。

This guide is about reading somebody else's evaluation write-up, not about replacing its conclusions.

An evaluation statement is a conditional sentence: who, on which task set, under which metric, compared with which version. Reading it properly means recovering all four parts.

This site runs no evaluations and reproduces no scores, and it ranks nothing across tiers. Capability statements on family cards each carry a source link; where there is no source, the field stays blank.

拿到一段评测说法，先找这四个成分，找不到就先记下缺口。

• 任务集范围：题目来自哪里、有多少类、训练与测试是否隔离。
• 指标口径：算的是通过率、精确匹配还是评分量表，分母是什么。
• 对照与版本：和什么比、对方用哪个版本、谁跑的评测。
• 声明出处：这句话出现在论文、模型卡还是发布公告，页面上是否仍可核对。

四个成分里只要有一个缺，这段说法就只能作为「未定口径」记录，不能升级成结论。

When you meet an evaluation claim, locate these four parts first and record any gap you cannot fill.

• Task-set scope: where the items come from, how many categories, whether training and test data are separated.
• Metric definition: pass rate, exact match or rating scale, and what the denominator counts.
• Comparison and version: against what, which release of the comparator, and who ran the evaluation.
• Statement location: whether the sentence appears in a paper, a model card or a release post, and whether that page is still checkable.

If any one of the four is missing, record the claim as an undefined measure rather than promoting it into a conclusion.

任务集本身定义了它能说明什么，也定义了它不能说明什么。

• MMLU 论文摘要写它是衡量文本模型 multitask accuracy 的测试，覆盖 57 项任务，出处见 arXiv 2009.03300。范围声明止于此，分数不外推到别的任务。
• Codex 论文摘要把 HumanEval 描述为衡量「由 docstring 生成程序」的功能正确性，出处见 arXiv 2107.03374。它测的是通过给定测试的能力，不是代码是否可维护。
• 同一个数字换分母就换含义：单题通过率、按任务宏平均、按样本数加权平均是三种口径。
• 任务集会被训练语料覆盖。是否隔离、怎么处理，要看该评测自己的说明。

A task set defines what it can say, and equally what it cannot.

• The MMLU abstract describes it as a test of a text model's multitask accuracy covering 57 tasks; source: arXiv 2009.03300. The scope statement stops there, and the score does not extend to other tasks.
• The Codex abstract describes HumanEval as measuring functional correctness for synthesizing programs from docstrings; source: arXiv 2107.03374. It tests passing given checks, not whether code is maintainable.
• The same figure means something else with another denominator: per-item pass rate, macro average across tasks and sample-weighted average are three different measures.
• Task sets can appear inside training corpora. Whether that was handled, and how, belongs to the evaluation's own description.

同一份权重在不同设置下会给出不同数字。下面这些项缺一项，可比性就缺一块。

• 提示模板与 few-shot 数量：模板不同，同一任务的得分可以不同。
• 上下文长度与截断规则：长输入被截时，得分测的是截断后的行为。
• 采样设置：贪心解码、温度、样本数与是否多次取样后取通过结果，都会改变得分。
• 评测脚本与版本：题目解析、判分实现和容错规则都随版本变。
• 对照组的算力与配置：对照模型是否用同一套设置跑，需要页面自己说明。

官方页没写这些项时，只有一条处理办法：官方未说明就按未说明记录，不替它补齐。

The same weights produce different numbers under different settings. Each missing item removes one piece of comparability.

• Prompt template and few-shot count: different templates can change the score on the same task.
• Context length and truncation rules: when long input is cut, the score describes the truncated behaviour.
• Sampling settings: greedy decoding, temperature, sample count and best-of-N all move results.
• Evaluation scripts and version: parsing, scoring implementation and tolerance rules change between releases.
• Comparator conditions: whether the baseline ran under the same setup has to be stated by the page itself.

When the official page omits these items there is one handling rule: record it as not stated and do not fill the gap with another model's settings.

三种口径的可信来源不同，读法也不同。

• 自报：作者自己跑并公布，通常最完整，也最需要看它公开的脚本与设置。
• 复现：第三方按公布方法重跑。差异要同时核对版本、设置与判分实现。
• 榜单：聚合口径常常统一了提示模板，也就统一不了你的任务。
• 厂商自评基准：本站在家族卡上会点名它是厂商自评，不当作横向裁决。

一句话口径：本站只说这句话有没有出处、出处覆盖了什么，不说它证明了模型有多好。

The three provenances carry different weight and are read differently.

• Self-reported: measured and published by the authors. Usually the most complete, so inspect the scripts and settings they expose.
• Replicated: a third party reruns the published method. Differences require checking versions, settings and scoring together.
• Leaderboards: aggregation often imposes one prompt template, which is not your task's template.
• Vendor self-assessment benchmarks: family cards name these explicitly as the vendor's own benchmark, not as a cross-vendor judgement.

The house wording is narrow: this site says whether a claim has a source and what that source covers. It never says the claim proves a model is good.

评测说明常带一句「在某种硬件上跑得动」，它同样只是条件句。

• QLoRA 论文摘要自述用 4-bit NormalFloat（NF4）这一数据类型，并自述在该方法下可在单张 48GB 显卡上微调 65B 参数模型，出处见 arXiv 2305.14314。这句话绑定的是论文里的方法与设置，不是任意机器的保证。
• Llama 3 论文摘要自述公开了 405B 参数的预训练与后训练版本，以及用于输入输出安全的 Llama Guard 3，出处见 arXiv 2407.21783。摘要本身不逐项列评测分数，分数要回到论文对应章节。
• 「单卡可跑」的口径要拆到精度、上下文长度、并发和运行时预留；这四个数不同，结论就不同。
• 参数量与体积都不能替你算出你这台机器的可用量，容量口径见「GPU 事实与成本数学」。

Evaluation write-ups often add a sentence about hardware. That sentence is conditional as well.

• The QLoRA abstract describes 4-bit NormalFloat (NF4) and reports fine-tuning a 65B parameter model on a single 48GB GPU; source: arXiv 2305.14314. The statement is bound to the method and settings in that paper, not a guarantee for any machine.
• The Llama 3 abstract describes releasing pretrained and post-trained 405B parameter models plus Llama Guard 3 for input and output safety; source: arXiv 2407.21783. The abstract itself does not itemise evaluation scores; those belong to the relevant paper sections.
• "Fits on one card" still needs precision, context length, concurrency and runtime reserve. Change those four and the conclusion changes.
• Neither parameter count nor file size computes the memory available on your machine; see "GPU facts and cost math".

下面的文字与数字都是虚构教学假设，不对应任何真实模型、官方页或本站运行结果。

假设某官方页写了一行数字：任务 A 得分 61.2，任务 B 得分 47.5。同一行还写着比上一版高 6.1。

• 先问分母：任务 A 有多少题、按什么判分；61.2 是通过率还是量表分。
• 再问设置：提示模板、few-shot 数量、上下文长度、采样与重复次数是否写明。
• 再问对照：上一版是否同脚本同硬件同配置，还是引用了旧页面里的数字。
• 最后问范围：这两个任务集是否覆盖你要做的事；不覆盖时这句话只说明它自己的范围。

如果四项都问不到答案，正确做法是把这行数字按原样记录，标记为口径未定。它不能被读成「这个模型更适合某任务」。

Everything in this example, including its numbers, is a fictional teaching assumption. It corresponds to no real model, official page or result produced by this site.

Suppose an official page said: "61.2 on task A, 47.5 on task B, 6.1 above the previous version."

• Ask about the denominator first: how many items task A has, how they are judged, and whether 61.2 is a pass rate or a scale score.
• Then ask about settings: prompt template, few-shot count, context length, sampling and repeat count.
• Then ask about the comparison: whether the previous version used the same script, hardware and configuration, or merely quoted an older page.
• Finally ask about scope: whether these two task sets cover what you intend to do. If they do not, the sentence only describes its own range.

If none of the four questions has an answer, record the numbers as published and mark the measure undefined. Do not read them as "this model suits a given task".

评测说明的价值在于给你一条可执行的检查路径，不是一个可以直接引用的结论。

• 先按自己的任务定判定标准，再决定要看哪些分数；判定标准可参考 怎么搭评测。
• 用固定输入跑一次结果检查，入口是 20 项自测清单。
• 需要记录假设与容量时，用 上下文与并发预算；它的算式只适用于同层标准全注意力 MHA / GQA，MLA、混合与滑动窗口要另算。
• 涉及费用构成时用 成本计算器，本站不保存任何单价。

学 AI 站的 模型评测的局限、基准测试 与 模型卡 是给使用者的一页科普。

这一篇给的是读官方评测说明时要补哪些条件。同一个概念在两处深度不同，互链不重复。

The value of an evaluation write-up is the check it lets you run, not a conclusion you can quote directly.

• Set your own task and pass criteria first, then decide which scores matter; criteria ideas are in building an evaluation.
• Run one fixed-input result check; the entry point is the 20-item self-test.
• To record assumptions and capacity, use the context and concurrency budget. Its formula applies only to standard same-layer full-attention MHA/GQA; MLA, hybrid and sliding-window designs need separate arithmetic.
• For cost composition use the cost calculator; this site stores no unit prices.

The Learn site pages limits of model evaluation, benchmark and model card are one-page explainers for users.

This guide lists which conditions to fill in when reading an official evaluation write-up. Same concept, different depth, cross-linked without duplication.