蒸馏：小模型怎么从大模型身上学Distillation: how small models learn from big ones从业者视角的方法论 · 关键事实挂官方出处practitioner-perspective methodology · key facts carry official sources

蒸馏 = 让小模型向大模型学，而不是直接从原始数据学。

原始方法是 Hinton 等人的软标签蒸馏（出处见文末清单）。
不只给「答案是猫」这个硬结论，还给出概率分布：
六成像猫、三成像狗、一成像狐狸。

类间相似度信息比硬标签的信息量大。
大模型时代，这条路分成了两支。

Distillation = a small model learns from a big one instead of learning straight from raw data.

The original method is Hinton et al.'s soft-label distillation (source in the list at the end).
It passes not just the hard verdict “the answer is cat” but the probability distribution:
six-tenths cat-like, three-tenths dog-like, one-tenth fox-like.

Inter-class similarity carries more information than a hard label.
In the LLM era this route split into two branches.

logit 蒸馏要同时跑教师与学生，教师输出分布当训练信号。

DistilBERT 是公开范例：同架构自蒸馏，参数减半、
速度提升 60%、保留 97% 语言理解能力（论文口径）。

开放权重家族里 logit 级蒸馏越来越少——教师太大，跑不起。

数据蒸馏只要教师生成数据，学生照常 SFT。
LLaMA 2 论文写明：13B 与 34B 用 70B 生成的对话数据训练。

DeepSeek-R1 把自家的长思维链样本蒸馏进一系列小模型，
从 1.5B 一直做到 70B，官方论文给出各尺寸的成绩表。
这是「思维链蒸馏」的官方范例。

MiniCPM 谱系把蒸馏写进训练流程的正式阶段（论文口径）。

Logits distillation runs teacher and student together, using the teacher's output distribution as the training signal.

DistilBERT is the public example: same-architecture self-distillation, parameters halved,
60% faster, retaining 97% of language understanding (the paper's figures).

Logit-level distillation is rarer and rarer among open-weight families — the teacher is too big to run.

Data distillation only needs the teacher to generate data; the student does ordinary SFT.
The LLaMA 2 paper states plainly: the 13B and 34B were trained on dialogue data generated by the 70B.

DeepSeek-R1 distilled its own long chain-of-thought samples into a series of small models,
from 1.5B up to 70B, with the official paper giving a results table per size.
This is the official example of “chain-of-thought distillation”.

The MiniCPM lineage writes distillation in as a formal stage of its training pipeline (the paper's wording).

三个理由，都有官方出处可查：

• 便宜：数据蒸馏不需要教师常驻显存，成本结构等于普通 SFT。

• 快：小模型推理便宜，跑在端侧与显卡农场里都划算（与预训练篇的「过训练」同一张账）。

• 许可：多数家族许可允许用其输出再训模型（以各许可文本为准，本站不解读条款）。

Three reasons, each with a checkable official source:

• Cheap: data distillation needs no teacher resident in memory — the cost structure equals ordinary SFT.

• Fast: small models are cheap to infer, paying off both on edge devices and GPU farms (the same ledger as “over-training” in the pre-training guide).

• Licence: most family licences allow re-training models on their outputs (each licence text governs; this site does not interpret terms).

本站口径与 S53 派生标记同一套纪律：
只有官方自己声明的谱系才标「蒸馏/派生」。
依据是模型卡的 base_model 字段，或论文、技术报告的明文。
不猜，也不根据「像」来推断。

事实层面的含义：蒸馏模型的能力上限受教师约束。
教师不会的，它大概率不会。

但小模型在窄任务上可以超过教师。
R1 蒸馏出的 7B 在数学评测上超过不少更大的通用模型（官方论文口径）。

「蒸馏来的」不是贬义标签。它告诉你权重谱系从哪来。
选型时把它当出处信息读，不当质量结论读。

This site follows the same discipline as the S53 derived-marker:
only lineages the vendor itself declares get marked “distilled/derived”.
The basis is the model card's base_model field, or explicit text in papers and technical reports.
No guessing, no inference from “looks similar”.

What it means factually: a distilled model's capability ceiling is bounded by its teacher.
What the teacher cannot do, it most likely cannot do.

But small models can beat the teacher on narrow tasks.
The 7B distilled from R1 outperforms several much larger general models on math evaluations (the official paper's figures).

“Distilled” is not a pejorative label. It tells you where the weight lineage comes from.
Read it as provenance when choosing, not as a quality verdict.

用某模型的输出去训新模型，是否受原模型许可约束？
各许可写法不同，Llama 与 Gemma 的条款各有附加约定（条款文本见文末清单）。

本站的立场与全站法律页一致：
只索引、只引述、不解读许可、不给法律意见。

Does training a new model on some model's outputs fall under the original model's licence?
Each licence is worded differently; Llama's and Gemma's terms each carry extra conditions (licence texts in the list at the end).

This site's stance matches the site-wide legal page:
index only, quote only, no licence interpretation, no legal advice.

一页科普版在学 AI 站：知识蒸馏
与 剪枝（压缩的另一条路）。

本篇讲两支蒸馏的成本结构、官方范例、选型读法与许可边界。
本站家族卡上的「派生自 X」标记（S53）用的就是本篇的谱系口径。

The one-page explainers live on the Learn site: knowledge distillation
and pruning (the other compression route).

This guide covers the cost structure of the two distillation branches, the official examples, how to read the label when choosing, and the licence boundary.
The “derived from X” marker on this site's family cards (S53) uses exactly this guide's lineage yardstick.