开源训练栈：多卡训练的工具链The open training stack: the toolchain behind multi-GPU training从业者视角的方法论 · 关键事实挂官方出处practitioner-perspective methodology · key facts carry official sources

模型一大，第一道坎不是算法，是装不下。
一张显存装不下参数、梯度和优化器状态，训练就必须拆到很多张卡上。

这一篇讲拆法的分类学，以及开源工具链各站在哪一层。
本站不实测，工具能力一律以官方仓库自述与论文口径为准，出处列在文末。

As a model grows, the first wall is not the algorithm — it does not fit.
When one GPU cannot hold parameters, gradients and optimiser states, training must be split across many cards.

This guide is the taxonomy of splitting methods and which layer each open toolchain sits at.
This site runs no benchmarks; tool capabilities follow the official repositories' self-descriptions and paper figures, sourced at the end.

数据并行：每张卡都有完整模型，各吃一批数据，梯度汇总。
简单，但模型本身必须装得下——这是天花板。

ZeRO 及其后代（FSDP）：模型仍按数据并行跑，
但把优化器状态、梯度、参数分片到各卡，用时再取回。
ZeRO 论文（arXiv 1910.02054）的核心洞察：数据并行的显存浪费是冗余，不是必需。
PyTorch 的 FSDP 是这条思路的官方实现（arXiv 2304.11277）。

张量并行：把单层矩阵乘法切给多张卡同时算。
通信最密，一般只在单机内 NVLink 互联的卡之间用。
原始方法是 Megatron-LM 论文（arXiv 1909.08053）。

流水线并行：按层切段，前向像流水线一样逐段传递。
通信少，但要注意「气泡」——流水线空转的时段。

大模型训练实际是三者叠加（业界俗称 3D 并行），
再按集群拓扑调各维度的大小。Llama 3 论文自报的并行配置就是这种组合。

Data parallelism: every card holds the full model, each chews a batch of data, gradients are aggregated.
Simple, but the model itself must fit — that is the ceiling.

ZeRO and its descendants (FSDP): the model still runs data-parallel,
but optimiser states, gradients and parameters are sharded across cards and fetched back when needed.
The core insight of the ZeRO paper (arXiv 1910.02054): data parallelism's memory waste is redundancy, not necessity.
PyTorch's FSDP is the official implementation of this idea (arXiv 2304.11277).

Tensor parallelism: one layer's matrix multiplications are cut across several cards computed simultaneously.
The most communication-dense; normally used only between cards linked by NVLink inside one machine.
The original method is the Megatron-LM paper (arXiv 1909.08053).

Pipeline parallelism: split by layer into stages, the forward pass moves stage by stage like an assembly line.
Less communication, but mind the “bubbles” — stretches where the pipeline idles.

Real large-model training stacks all three (industry shorthand: 3D parallelism),
sizing each dimension to the cluster topology. The parallel configuration the Llama 3 paper self-reports is exactly such a combination.

DeepSpeed（微软官方仓库）：通用分布式训练优化库，ZeRO 系列的落地处。

Megatron-LM（NVIDIA 官方仓库）：规模化 transformer 训练框架，
张量并行的参考实现，大量开放权重模型的训练底座。

torchtitan（PyTorch 官方仓库）：PyTorch 原生的训练平台，
把 FSDP、张量并行、流水线并行作为原生组合提供。

nanotron（Hugging Face 官方仓库）：极简的 3D 并行训练库，
官方自述就是「minimalistic」，读代码学并行策略的入口。

TRL（Hugging Face 官方仓库）：站在后训练层——SFT、奖励模型、
DPO、RLHF 的工具链，接在预训练框架之后用。

选型判断依据（不是排名）：想读懂并行原理，从 nanotron 或 torchtitan 入手；
要复现大模型训练，看目标模型的官方训练说明用了什么；
只做后训练微调，TRL 加 FSDP 通常够——预训练级并行根本用不上。

DeepSpeed (Microsoft's official repository): the general distributed-training optimisation library, where the ZeRO line landed.

Megatron-LM (NVIDIA's official repository): the scaled transformer training framework,
the reference implementation of tensor parallelism, the training base under many open-weight models.

torchtitan (PyTorch's official repository): PyTorch-native training platform,
offering FSDP, tensor and pipeline parallelism as native combinations.

nanotron (Hugging Face's official repository): a minimal 3D-parallel training library,
self-described by the team as “minimalistic” — the entry point for learning parallel strategies by reading code.

TRL (Hugging Face's official repository): sits at the post-training layer — the toolchain for SFT, reward models,
DPO and RLHF, used after the pre-training framework.

Judgement basis for selection (not a ranking): to understand parallelism, start from nanotron or torchtitan;
to reproduce large-model training, see what the target model's official training notes say it used;
for post-training fine-tuning only, TRL plus FSDP is usually enough — pre-training-grade parallelism never comes up.

把框架当配方。 同一个框架换个并行配置，MFU 能差出几倍。
框架解决「能不能跑」，调优决定「跑得多贵」。

低估通信。 张量并行跨机跑，带宽墙会吃掉论文里看不到的性能。
各家论文自报的并行维度分配（哪些维度在机内、哪些跨机）值得照抄结构、不照抄数字。

把训练栈与推理栈混为一谈。 训练框架管「怎么造」，
vLLM 这类推理引擎管「怎么跑」，优化目标完全不同。

拿开源栈的数字对标闭源大厂。 大厂集群的网络拓扑、故障恢复、
检查点策略都是自研的。开源栈给你的是同量级的方法，不是同量级的利用率。

Mistaking the framework for the recipe. The same framework with a different parallel configuration can differ several-fold in MFU.
Frameworks decide “can it run”; tuning decides “how expensive it runs”.

Underestimating communication. Tensor parallelism across machines hits a bandwidth wall that eats performance invisible in papers.
The parallel-dimension layouts papers self-report (which dimensions stay in-machine, which cross machines) are worth copying as structure, not as numbers.

Confusing the training stack with the inference stack. Training frameworks manage “how to build it”;
vLLM-class inference engines manage “how to run it” — the optimisation targets are completely different.

Benchmarking open stacks against closed giants. Big labs' cluster network topology, failure recovery and
checkpointing strategies are all in-house. Open stacks give you methods of the same order, not utilisations of the same order.

学 AI 站的 张量并行 与
流水线并行 是使用者视角的一页科普；
这一篇是从业者的选型地图：各工具站在哪一层、坑在哪。

同一个概念在两处深度不同，互链不重复。
语料与算力的记账方法见上一篇「数据与算力」。

The Learn site's tensor parallelism and
pipeline parallelism pages are user-perspective explainers;
this guide is the practitioner's selection map: which layer each tool sits at, and where the traps are.

Same concept, different depth on each side, cross-linked without duplication.
For the bookkeeping method of corpora and compute see the previous guide, “Data and compute”.