偏好对齐：RLHF 与 DPO 在解决什么问题Preference alignment: what RLHF and DPO actually solve从业者视角的方法论 · 关键事实挂官方出处practitioner-perspective methodology · key facts carry official sources

对齐解决的是「回答得好」，不是「回答得出」。

SFT 之后模型会按格式答，但「哪个答案更好、哪里越界」没有信号。
这一步的原始流程是 InstructGPT（arXiv 2203.02155）的 RLHF：
人给回答排序，训一个奖励模型，再用 PPO 让语言模型去讨好这个奖励模型。

Alignment solves “answering well”, not “being able to answer”.

After SFT the model answers in format, but there is no signal for “which answer is better, where the line is crossed”.
The original recipe for this step is InstructGPT's (arXiv 2203.02155) RLHF:
humans rank answers, a reward model is trained, then PPO makes the language model court that reward model.

人不直接打分，而是成对比较：同一提示的两个回答，选哪个更好。
Anthropic 的 HH 数据（arXiv 2204.05862）把维度拆成 helpful 与 harmless 两条。
奖励模型学的就是这个排序倾向，输出一个标量分。

从业者要盯的坑：奖励模型是代理指标，不是好坏本身。
它能被讨好——回答更长、更自信、结构更像范文，分就高。
「长度偏差」是 RLHF 时代反复被官方论文承认的副作用。

Humans do not score directly; they compare pairs: of two answers to the same prompt, which is better.
Anthropic's HH data (arXiv 2204.05862) splits the axes into helpful and harmless.
The reward model learns this ranking tendency and emits a scalar score.

The trap practitioners must watch: the reward model is a proxy metric, not goodness itself.
It can be gamed — longer, more confident, model-essay-like answers score higher.
“Length bias” is a side effect repeatedly acknowledged in RLHF-era papers.

InstructGPT 的 RL 阶段要同时驻留四个模型：
被训的策略模型、防跑偏的参考模型（KL 约束）、奖励模型、critic。
显存与算力开销约是 SFT 的数倍，调参敏感（KL 系数一歪就崩）。

LLaMA 2（arXiv 2307.09288）同时给了两条路线对照：
SFT 循环自举，对比拒绝采样加 PPO，各自在不同评测上互有胜负。
官方自己也没宣布一边完胜，本站照实转述。

InstructGPT's RL stage keeps four models resident at once:
the policy being trained, the reference model that keeps it on the rails (KL constraint), the reward model, and the critic.
Memory and compute cost several times SFT, and tuning is touchy (a skewed KL coefficient collapses the run).

LLaMA 2 (arXiv 2307.09288) published both routes side by side:
iterative SFT bootstrapping versus rejection sampling plus PPO, each winning on different evaluations.
The authors themselves declared no outright winner; this site reports it as stated.

DPO（arXiv 2305.18290）的推导说明：最优策略有闭式解，
可以直接拿偏好对做分类式损失，不用训奖励模型、不用跑 RL。
显存回到 SFT 量级，代码几十行。

Zephyr（arXiv 2310.16944）用公开数据把 7B 做到当时同规模前列，
证明这条路线社区可复现。

代价也要说：DPO 对偏好对的分布敏感，离线数据训容易「过拟合排序风格」；
在线 RLHF 能持续采样新回答，信号更新鲜。
两条路线的公开对照结论不稳定，因数据而异。
本站不裁决哪条更好，只记录各自的机制与已发表的边界。

DPO's (arXiv 2305.18290) derivation shows the optimal policy has a closed form,
so preference pairs can feed a classification-style loss directly — no reward model to train, no RL to run.
Memory drops back to SFT scale, the code is dozens of lines.

Zephyr (arXiv 2310.16944) took a 7B model to the front of its size class at the time using public data,
proving the route is community-reproducible.

The cost too: DPO is sensitive to the distribution of preference pairs, and offline training easily “overfits the ranking style”;
online RLHF keeps sampling fresh answers, so the signal stays fresher.
Published head-to-head conclusions between the two routes are unstable and data-dependent.
This site adjudicates neither — it records each mechanism and its published limits.

Constitutional AI（arXiv 2212.08073）用一组书面原则让模型自我批评、
自我修订，再由模型偏好模型。人工排序退到校准位。

开放权重家族的后训练普遍混用这条路——
规模上不去的瓶颈从算力转到标注。

Constitutional AI (arXiv 2212.08073) uses a written set of principles to make the model criticise and
revise itself, then lets models prefer models. Human ranking retreats to a calibration role.

Open-weight families' post-training routinely mixes this route in —
the bottleneck that caps scale moves from compute to annotation.

「对齐税」指对齐后模型在原始任务评测上可能掉分。
InstructGPT 论文自己观察到 GPT-3 在部分公开基准上因对齐回落。

谄媚（sycophancy）是偏好数据的镜像产物：
人偏爱「顺着我说」的回答，奖励模型就学会顺着说。

这两条都不是传闻，是论文正文承认的副作用。
读任何模型的官方 benchmark 时都要记得：
对齐后的分数是「讨好评测的风格」与「真实能力」的混合量。

The “alignment tax” is the score a model can lose on original-task evaluations after alignment.
The InstructGPT paper itself observed GPT-3 regressing on some public benchmarks because of alignment.

Sycophancy is the mirror image of preference data:
humans favour answers that agree with them, so the reward model learns to agree.

Neither is rumour — both are side effects admitted in the papers themselves.
When reading any model's official benchmarks, remember:
post-alignment scores mix “a style that pleases the grader” with “actual capability”.

一页科普版在学 AI 站：训练三阶段
与 偏好对齐。

本篇讲四模型驻留的成本结构、代理指标可被讨好的机制、
对齐税与谄媚的出处——从业者要盯的坑，一页科普装不下。

The one-page explainers live on the Learn site: three-stage training
and preference alignment.

This guide covers the cost structure of four resident models, how a proxy metric can be gamed,
and the sources behind the alignment tax and sycophancy — the traps practitioners watch, which an explainer page cannot hold.