跑起来深水：KV 缓存、批处理与推理引擎Serving deep dive: KV cache, batching, and inference engines从业者视角的方法论 · 关键事实挂官方出处practitioner-perspective methodology · key facts carry official sources

模型跑起来之后，吞吐的瓶颈多半不在算力，在显存里的 KV 缓存。
这一篇讲推理引擎围绕这块显存做的三件事：怎么存、怎么批、怎么复用。

本站不实测。所有倍数都是论文自报口径，出处列在文末；
引擎能力一律以官方文档为准。

Once a model is running, the throughput bottleneck is usually not compute but the KV cache in memory.
This guide covers the three things inference engines do around that memory: how to store, how to batch, how to reuse.

This site runs no benchmarks. Every multiplier is a paper's self-reported figure, sourced at the end;
engine capabilities always follow the official docs.

自回归生成每产出一个 token，都要回看前文。
为避免重复计算，前文的 key/value 张量被缓存下来，这就是 KV 缓存。

它的显存随「上下文长度 × 并发请求数」线性增长，而且动态伸缩。
PagedAttention 论文指出：低效管理下，碎片与冗余复制会显著浪费这块显存，
反过来限制批大小——吞吐上不去，往往不是算不动，是塞不进。

判断依据：同一张卡，长上下文与高并发同时上来时吞吐骤降，
多半就是 KV 缓存显存先到顶。

Autoregressive generation revisits the preceding text for every token it produces.
To avoid recomputation, the earlier key/value tensors are cached — that is the KV cache.

Its memory grows linearly with “context length × concurrent requests” and scales dynamically.
The PagedAttention paper points out that under inefficient management, fragmentation and redundant copies waste this memory badly,
which in turn caps batch size — throughput stalls not because compute ran out but because requests won't fit.

Rule of thumb: on the same card, throughput collapses when long context and high concurrency arrive together —
most likely the KV-cache memory hit its ceiling first.

PagedAttention 借操作系统虚拟内存的分页思路：
KV 缓存按固定大小的块分配，不要求连续。论文自报近零浪费。
vLLM 在其上构建，论文自报同等延迟下吞吐提升 2–4 倍——注意这是论文实验口径。

连续批处理解决另一件事：请求到达时间不齐。
静态批要等最慢的完成才能换下一批；连续批让完成的请求随时退出、新请求随时补位。
vLLM 官方文档把两者作为吞吐来源并列描述。

分块预填充是配套的第三件：长提示的预填充阶段会独占 GPU，
把它切成块与解码步交替，延迟曲线更平。

PagedAttention borrows the paging idea from operating-system virtual memory:
the KV cache is allocated in fixed-size blocks that need not be contiguous. The paper self-reports near-zero waste.
vLLM builds on it; the paper self-reports 2–4× throughput at equal latency — note this is the paper's experimental setting.

Continuous batching solves a different problem: requests arrive unevenly.
A static batch waits for the slowest request before swapping in the next batch; continuous batching lets finished requests leave at any time and new ones slot in.
vLLM's official docs describe both side by side as throughput sources.

Chunked prefill is the companion third move: a long prompt's prefill phase hogs the GPU,
so it is cut into chunks interleaved with decode steps, flattening the latency curve.

vLLM：通用 LLM 服务引擎，PagedAttention 的官方落地，
文档覆盖量化、并行、多机部署。开放权重部署讨论里的默认名字。

SGLang：面向「多调用程序」的服务框架——代理循环、推理链、结构化输出这类负载。
论文自报两项机制：RadixAttention 复用跨调用共享前缀的 KV 缓存；
压缩有限状态机加速结构化输出解码。论文自报此类任务上最高 6.4 倍吞吐。

两者不是排名关系，负载画像不同：
单轮问答、批量离线生成，vLLM 路线成熟；
多轮代理、共享长前缀、严格格式输出，SGLang 的复用机制对症。
本站不裁决，论文口径都在文末。

本地单机跑 GGUF 是另一条线：llama.cpp 一类实现不追求并发吞吐，
追求在有限显存里把模型装下。选型先问「几个并发」，再问引擎。

vLLM: the general-purpose LLM serving engine, the official landing of PagedAttention,
with docs covering quantisation, parallelism and multi-node deployment. The default name in open-weight deployment discussions.

SGLang: a serving framework aimed at “multi-call programs” — agent loops, reasoning chains, structured-output workloads.
The paper self-reports two mechanisms: RadixAttention reuses KV cache across shared prefixes between calls;
compressed finite-state machines speed up structured-output decoding. The paper self-reports up to 6.4× throughput on such tasks.

The two are not a ranking — their workload profiles differ:
for single-turn Q&A and batch offline generation, the vLLM route is mature;
for multi-turn agents, shared long prefixes and strict-format output, SGLang's reuse mechanisms fit.
This site adjudicates neither; the paper figures are all at the end.

Running GGUF locally on one machine is a separate line: llama.cpp-style implementations do not chase concurrent throughput,
they chase fitting the model into limited memory. Ask “how many concurrent requests” first, then ask about the engine.

解码每步只产一个 token，GPU 算力大量闲置。
投机解码让小模型先猜几个 token，大模型一次并行验证；猜对即白赚。

它改的是单请求延迟，不是吞吐账；猜错率高的负载收益缩水。
vLLM 与 SGLang 官方文档都提供该选项，收益取决于草稿模型与任务匹配度。

Decoding produces one token per step, leaving most GPU compute idle.
Speculative decoding has a small model guess several tokens first while the big model verifies them in one parallel pass; correct guesses are free profit.

It changes single-request latency, not the throughput ledger; workloads with high miss rates see the gains shrink.
Both vLLM and SGLang official docs offer the option; the payoff depends on how well the draft model matches the task.

学 AI 站的 KV 缓存、
PagedAttention、
连续批处理 与
投机解码 是使用者视角的一页科普；
这一篇讲从业者怎么给推理记账、怎么按负载画像选引擎。

同一个概念在两处深度不同，互链不重复。
硬件事实与成本数学见下一篇「GPU 事实与成本数学」。

The Learn site's KV cache,
PagedAttention,
continuous batching and
speculative decoding pages are user-perspective explainers;
this guide is about how practitioners keep the inference ledger and pick an engine by workload profile.

Same concept, different depth on each side, cross-linked without duplication.
For hardware facts and cost math see the next guide, “GPU facts and cost math”.