数据与算力：做模型的两大开销Data and compute: the two big bills of making a model从业者视角的方法论 · 关键事实挂官方出处practitioner-perspective methodology · key facts carry official sources

做模型的钱主要花在两处：数据与算力。
这一篇讲从业者怎么给这两笔开销记账：语料从哪来、洗到什么规模；
算力用什么单位自报、利用率实际有多高；成本量级怎么估。

本站不实测，也不收录任何具体价格。
所有数字都是论文或官方报告自报的口径，出处列在文末。

The money in making a model goes to two bills: data and compute.
This guide is about how practitioners keep the books on these two costs: where corpora come from and what scale they get washed down to;
what units papers self-report compute in and how high utilisation actually runs; how to estimate cost magnitude.

This site runs no benchmarks and stores no specific prices.
Every figure is a self-reported number from a paper or official report, sourced at the end of the page.

预训练语料的主力来源是 Common Crawl——
一家非营利组织维护的免费开放网页存档，每月发布一批 dump。

原始网页是脏的。同一篇文章在几十个站点上有几十个副本。
导航栏、模板文本、机器生成内容混在正文里。
清洗管线大致是：抽正文、按语言筛、启发式过滤、去重。

去重是整条管线里最贵的步骤之一。
Common Crawl 各批 dump 之间互相重叠。
跨批去重要在几十万亿 token 上做相似度匹配。
FineWeb 报告把每一步的取舍都做了消融，公开可查。

The workhorse source of pre-training corpora is Common Crawl —
a free, open web archive maintained by a non-profit, publishing a batch of dumps every month.

Raw web pages are dirty. The same article exists in dozens of copies across dozens of sites.
Navigation bars, template text and machine-generated content are mixed into the body.
The cleaning pipeline is roughly: extract the body, filter by language, heuristic filters, deduplicate.

Deduplication is one of the most expensive steps in the whole pipeline.
Common Crawl's dump batches overlap each other.
Cross-batch deduplication means similarity matching over tens of trillions of tokens.
The FineWeb report ablates every step's trade-off, publicly checkable.

头部模型的语料配方多数不公开。公开语料的价值就在这里：
让没预算攒数据的人也能训出能用的模型。

FineWeb（Hugging Face）：15T token，来自 96 批 Common Crawl 存档。
另附教育向子集 FineWeb-Edu，1.3T token。

DCLM（普林斯顿）：语料标准化到 240T token，训练预算固定。
参与者只比一项：数据筛选方法。结论直白：多喂不如喂对。

Dolma（AI2）：3T token 英文研究语料，明确定位是给科研用的对照底座。

三家都把清洗代码一起放出。选开放语料时看许可与清洗记录，
别只看 token 总数——总数大、洗得糙，训出来照样差。

Most frontier models keep their corpus recipe private. That is exactly the value of open corpora:
people without a budget to assemble data can still train usable models.

FineWeb (Hugging Face): 15T tokens, from 96 Common Crawl dump batches.
Plus the education-oriented subset FineWeb-Edu at 1.3T tokens.

DCLM (Princeton): the corpus standardised to 240T tokens with a fixed training budget.
Participants compete on one variable only: the data-filtering method. The conclusion is blunt: feeding right beats feeding more.

Dolma (AI2): a 3T-token English research corpus, explicitly positioned as a control base for research.

All three release their cleaning code too. When choosing an open corpus, look at the licence and the cleaning record —
not just the token total: a big total washed carelessly still trains badly.

训练算力通行的记账单位是「卡时」：一张加速卡跑一小时。
论文自报口径各不相同，比较前先看清是什么卡。

DeepSeek-V3 论文自报：完整训练用了 2.788M 个 H800 卡时。
Llama 3 论文自报：405B 模型最多用 16K 块 H100 训练。
PaLM 论文自报：6144 块 TPU v4，训练 FLOPs 约 2.56×10²⁴。
三篇论文的编号都在文末出处清单里。

卡与卡不同代、不同厂商，卡时不能直接互比。
跨硬件比较用 FLOPs（浮点运算次数）更公平，但论文未必都报。

MFU（模型算力利用率）是另一个关键口径：
模型真正用到的运算，除以硬件理论峰值。
Llama 3 论文自报 MFU 38–43%。
也就是说，一半以上硬件时间花在通信、重算与故障恢复上。
看别人晒的「卡数」时，先问 MFU，不然会严重低估真实开销。

The common bookkeeping unit for training compute is the GPU-hour: one accelerator running for one hour.
Papers self-report in different terms — check which card before comparing.

The DeepSeek-V3 paper self-reports: full training took 2.788M H800 GPU-hours.
The Llama 3 paper self-reports: the 405B model trained on up to 16K H100s.
The PaLM paper self-reports: 6,144 TPU v4 chips, training FLOPs about 2.56×10²⁴.
The paper numbers are all in the source list at the end.

Cards differ by generation and vendor; GPU-hours are not directly comparable across them.
Across hardware, FLOPs (floating-point operations) is the fairer unit, though papers do not always report it.

MFU (model FLOPs utilisation) is the other key yardstick:
the computation the model actually uses, divided by the hardware's theoretical peak.
The Llama 3 paper self-reports MFU of 38–43%.
That is, more than half of hardware time goes to communication, recomputation and failure recovery.
When someone shows off a “card count”, ask about MFU first — otherwise you badly underestimate the real cost.

千卡级训练里，硬件故障不是意外，是日常。
Llama 3 论文自报：预训练 54 天快照期内共 466 次任务中断，
其中 419 次是意外中断，约 78% 归因于硬件问题；
即便如此，有效训练时间仍超过 90%。

这意味着算力账单里含一大块「白烧」：
故障回滚掉的重训、检查点保存、为稳定性预留的冗余。
DeepSeek-V3 摘要自报：全程没有不可恢复的 loss spike，也没有回滚。
这句话被当成卖点写进摘要，说明它在业界是稀缺事件。

架构选择也能改这笔账。GLaM 论文自报：
模型 1.2 万亿参数，训练能耗约为 GPT-3 的三分之一，推理 FLOPs 约一半。
稀疏激活省的是「每个 token 激活的参数量」，不是参数量本身。

At thousand-card scale, hardware failures are not accidents; they are routine.
The Llama 3 paper self-reports: across the 54-day pre-training snapshot there were 466 job interruptions,
419 of them unexpected, roughly 78% attributed to hardware issues;
even so, effective training time stayed above 90%.

That means the compute bill contains a large “burned for nothing” slice:
retraining rolled back after failures, checkpoint saving, redundancy reserved for stability.
The DeepSeek-V3 abstract self-reports: no unrecoverable loss spike and no rollback across the whole run.
The fact being written into the abstract as a selling point tells you how rare it is in industry.

Architecture choices can change this bill too. The GLaM paper self-reports:
a 1.2-trillion-parameter model whose training energy was about a third of GPT-3's, with inference FLOPs about half.
Sparse activation saves “parameters activated per token”, not the parameter count itself.

训练成本的构成是一条乘法链：
卡时数 × 每卡每小时的租用价格 ≈ 训练成本。

第一项论文会自报（见上节）；第二项随市场每月波动，
本站按纪律不收录任何具体单价，也不做「训一个模型要多少钱」的报价。

能诚实给出的是量级判断方法：
先查同规模模型的自报卡时；再按 Chinchilla 口径核对训练量，
即每参数约 20 个 token；最后用 MFU 修正硬件利用率。
三步走完，你至少知道这笔账是十万卡时级还是千万卡时级。
这已经足够支撑「要不要自己做模型」的判断。

Training cost decomposes into a multiplication chain:
GPU-hours × hourly rental price per card ≈ training cost.

The first term papers self-report (see the previous section); the second moves with the market every month,
and this site, by discipline, stores no unit prices and issues no “how much it costs to train a model” quotes.

What can be given honestly is a method for judging magnitude:
first look up self-reported GPU-hours of same-scale models; then check the training volume against the Chinchilla yardstick,
about 20 tokens per parameter; finally correct for hardware utilisation with MFU.
Finish those three steps and you at least know whether the bill is in the hundred-thousand-GPU-hour or ten-million-GPU-hour range.
That is already enough to support the judgement of “should we make our own model”.

学 AI 站的 训练语料、
训练算力 与
AI 芯片 是使用者视角的一页科普；
这一篇讲从业者怎么记账：语料管线、自报口径、利用率与故障成本。

同一个概念在两处深度不同，互链不重复。
把并行训练落地的开源工具链见本站「开源训练栈」篇。

The Learn site's training corpora,
training compute and
AI chips pages are user-perspective explainers;
this guide is about how practitioners keep the books: corpus pipelines, self-reported yardsticks, utilisation and failure costs.

Same concept, different depth on each side, cross-linked without duplication.
For the open toolchain that makes parallel training real, see this site's “The open training stack” guide.