预训练：模型怎么做到「什么都知道一点」Pre-training: how a model comes to know a bit of everything从业者视角的方法论 · 关键事实挂官方出处practitioner-perspective methodology · key facts carry official sources

预训练就是让模型在海量文本上学会续写。
学 AI 站有一页科普讲「它是什么」；这一篇讲从业者要做的决定：
语料怎么攒、算力怎么配、训练量怎么定、词表怎么切。

关键事实都有公开的论文或官方模型卡做出处，
出处列在文末；本站只转述与对照，不实测。

Pre-training is making a model learn to continue text on massive amounts of it.
The Learn site has an explainer page on “what it is”; this guide is about the decisions practitioners make:
how corpora are assembled, how compute is budgeted, how training volume is set, how vocabularies are cut.

Every key fact has a public paper or official model card as its source,
listed at the end of the page; this site only transcribes and cross-checks — it runs no benchmarks.

主流大模型的目标单一到乏味：给定前文，预测下一个 token 的概率分布。

GPT-3 论文（arXiv 2005.14165）把这条路线推到 1750 亿参数，
证明规模够大之后不用专门教，能力会自己长出来。
BERT 那条「完形填空」路线（双向掩码）在生成任务上输了。
此后开放权重家族几乎全是自回归解码器结构。

从业者视角的推论：模型「知道」的东西，全是续写统计的副产品。
它没有为任何任务专门学过。这解释了它为什么通用，
也解释了它为什么会一本正经地编。

The objective behind mainstream LLMs is boringly single: given the preceding text, predict the probability distribution of the next token.

The GPT-3 paper (arXiv 2005.14165) pushed this route to 175 billion parameters and
showed that at sufficient scale abilities grow on their own without being taught explicitly.
BERT's “fill-in-the-blank” route (bidirectional masking) lost on generation tasks.
Since then, open-weight families are almost all autoregressive decoder architectures.

Practitioner corollary: everything a model “knows” is a by-product of continuation statistics.
It was never trained for any specific task. That explains why it is general,
and also why it fabricates with a straight face.

网页语料（Common Crawl）是脏的：重复、垃圾页、机器生成文本混在里面。
T5 论文的 C4 附录给出了一套被广泛沿用的清洗规则：
去重、过滤低质量行、按语言筛。

LLaMA 论文明确说只用公开可得数据训练，把「数据配方」本身当成贡献。

清洗比规模更关键这件事，有官方对照实验。
普林斯顿的 DCLM 报告（arXiv 2406.11794）用同一套训练预算只换数据筛选方法，
模型表现差距巨大——多喂不如喂对。

Hugging Face 的 FineWeb（arXiv 2406.17557）把这套清洗做成 15T token 的开放语料，
每步清洗的消融都公开，可以直接复现。

去重是硬功夫：同一篇网文在 Common Crawl 里可能出现几十次。
重复数据不仅浪费算力，还会让模型把训练文本原样背下来。
这是评测污染的一大来源。

Web corpora (Common Crawl) are dirty: duplicates, junk pages and machine-generated text are mixed in.
The T5 paper's C4 appendix set out widely followed cleaning rules:
deduplicate, filter low-quality lines, filter by language.

The LLaMA paper states plainly that it trained only on publicly available data, treating the “data recipe” itself as a contribution.

That curation matters more than scale has a controlled experiment behind it.
Princeton's DCLM report (arXiv 2406.11794) held the training budget fixed and changed only the data-filtering method —
model performance varied enormously. Feeding more loses to feeding right.

Hugging Face's FineWeb (arXiv 2406.17557) turned this cleaning pipeline into an open 15T-token corpus,
with an ablation published for every cleaning step, directly reproducible.

Deduplication is hard graft: the same web article can appear dozens of times inside Common Crawl.
Duplicates waste compute and make the model memorise training text verbatim —
a major source of benchmark contamination.

2020 年 Kaplan 等人（arXiv 2001.08361）给出第一版缩放律：
损失随参数量、数据量、算力平滑下降，参数优先——钱该花在更大的模型上。

2022 年 Chinchilla（arXiv 2203.15556）用更细的实验推翻了这个配比：
算力固定时，参数和数据要一起长，最优大约是每参数 20 个 token。
按旧口径训的大模型，很多是「喂少了」的。

这个结论直接改了开放权重的路线。
LLaMA 系列（7B 喂 1.0T、70B 喂 1.4T token）走的就是「小模型 + 超量数据」。
Llama 3.1 官方模型卡写明 15T token——按 Chinchilla 口径这是远超最优的「过训练」。

为什么明知「过训练」还要过训练？因为缩放律优化的是训练成本，
而模型要反复推理。小模型多喂数据，训练稍贵、推理便宜得多。
开放权重模型要跑在千万用户的显卡上，这个账必须这么算。

In 2020 Kaplan et al. (arXiv 2001.08361) gave the first scaling laws:
loss falls smoothly with parameters, data and compute — parameters first, spend the money on bigger models.

In 2022 Chinchilla (arXiv 2203.15556) overturned that budget with finer experiments:
at fixed compute, parameters and data should grow together, optimally about 20 tokens per parameter.
Many big models trained under the old consensus were “under-fed”.

This result directly rerouted open weights.
The LLaMA line (1.0T tokens for 7B, 1.4T for 70B) is exactly “small model + over-abundant data”.
The Llama 3.1 official model card states 15T tokens — by Chinchilla's yardstick a far-past-optimal “over-training”.

Why over-train knowingly? Because scaling laws optimise training cost,
while the model will be inferred against repeatedly. Feed a small model more data: training costs a bit more, inference gets much cheaper.
Open-weight models have to run on tens of millions of users' GPUs — the math has to be done this way.

一次预训练不是一根直线。学习率先升后降（常用余弦退火），
末期常在高质量数据上「退火」收尾。各家具体配方多数不公开，
本站只写官方公开过的部分，没公开的就不写。

上下文长度是训练后再拉长的，这是常见做法。
LLaMA 2 论文写明用位置插值把 4K 扩到 32K；Llama 3.1 官方模型卡标 128K。
拉长靠改 RoPE 位置编码的频率基数等手法。

代价要如实说：「训练时见过的长度」和「标称长度」能力不等。
标 128K 不等于 128K 处还能准确找针，评测要自己按长度分档看。

A pre-training run is not one straight line. The learning rate ramps up then down (cosine annealing is common),
and the final phase often “anneals” on high-quality data. Most families keep their exact recipe private;
this site writes only what has been officially published and leaves the rest unwritten.

Context length is usually stretched after training — standard practice.
The LLaMA 2 paper states it extended 4K to 32K with position interpolation; the Llama 3.1 model card lists 128K.
Stretching relies on techniques such as changing the frequency base of RoPE positional encoding.

State the cost honestly: “length seen during training” and “nominal length” are not equal in ability.
A 128K label does not mean needle-in-a-haystack still works accurately at 128K — evaluate by length buckets yourself.

模型看到的不是字，是 token。主流切法是 BPE（arXiv 1508.07909）：
从字符起步，把最常共现的片段逐步合并成子词。

词表大小是个权衡：表大，中文这类非英语文本更省 token；
表小，切得碎、序列变长。
Llama 用 128K 词表（官方模型卡），OpenAI 的 o200k_base 到 20 万（tiktoken 官方仓库）。

同一句话在不同词表下 token 数能差几倍。
比较「上下文长度」「单价」之前，先确认词表口径，不然数字没有可比性。

Models do not see characters, they see tokens. The mainstream splitter is BPE (arXiv 1508.07909):
start from characters and iteratively merge the most co-occurring fragments into subwords.

Vocabulary size is a trade-off: a big table saves tokens on non-English text like Chinese;
a small table cuts finer and sequences get longer.
Llama uses a 128K vocabulary (official model card); OpenAI's o200k_base reaches 200K (tiktoken official repository).

The same sentence can differ several-fold in token count across vocabularies.
Before comparing “context length” or “price per token”, check the vocabulary yardstick — otherwise the numbers are not comparable.

学 AI 站的 预训练 与
训练三阶段 是使用者视角的一页科普；
这一篇是从业者视角的深版：讲决定、讲代价、讲口径变化。

同一个词在两处深度不同，互链不重复。
数据与算力的成本量级见本站[数据与算力篇](data-compute.html)，
开源训练栈见[开源训练栈篇](training-stack.html)。

The Learn site's pre-training and
three-stage training pages are user-perspective explainers;
this guide is the practitioner deep version: decisions, costs, and how the consensus changed.

Same term, different depth on each side, cross-linked without duplication.
For the cost magnitude of data and compute see this site's [Data and compute guide](data-compute.html);
for the open training stack see [The open training stack](training-stack.html).