GPU 事实与成本数学：跑模型的硬件账GPU facts and cost math: the hardware bill for running a model从业者视角的方法论 · 关键事实挂官方出处practitioner-perspective methodology · key facts carry official sources

「我的硬件能跑多大的模型」是一道可以手算的题。
这一篇给出硬件侧的关键事实（只录官方规格页写明的数字），
再给出成本数学的构成方法。

本站按纪律不收录任何卡价与租用单价，也不做购卡推荐。
数字随厂商改版会变，复核日期见页脚。

“How big a model can my hardware run” is a problem you can do by hand.
This guide gives the hardware-side key facts (only figures stated on official spec pages),
then the composition method for the cost math.

By discipline this site stores no card prices or rental rates and gives no purchase advice.
Figures change when vendors revise; the re-check date is in the footer.

判断「能不能跑」看两个数：显存容量决定装得下多大的模型，
显存带宽决定 decode 阶段每秒能吐多少 token。

- NVIDIA H100 SXM：显存 80GB，带宽 3.35TB/s（官方规格页）。
- NVIDIA H100 NVL：显存 94GB，带宽 3.9TB/s（同一页两列对照）。
- NVIDIA A100 80GB：HBM2e，带宽 1,935 GB/s（PCIe）/ 2,039 GB/s（SXM）。
- NVIDIA RTX 4090：24GB GDDR6X（消费卡，官方页未标带宽数值）。
- Apple MacBook Pro（M5 系列）：统一内存 16/24/32GB 起配，
高配可到 128GB——CPU 与 GPU 共享同一块内存（官方规格页）。

诚实口径：AMD 消费卡规格页本次未能核验（站点不可达），
本站就不录 AMD 数字——未知就说未知，不抄二手参数表。
各卡功耗、互联、FP8 支持等细节以官方页为准，此处不展开。

Two numbers decide “can it run”: memory capacity decides how big a model fits,
and memory bandwidth decides how many tokens per second the decode phase can emit.

- NVIDIA H100 SXM: 80GB memory, 3.35TB/s bandwidth (official spec page).
- NVIDIA H100 NVL: 94GB memory, 3.9TB/s bandwidth (both columns on the same page).
- NVIDIA A100 80GB: HBM2e, 1,935 GB/s (PCIe) / 2,039 GB/s (SXM) bandwidth.
- NVIDIA RTX 4090: 24GB GDDR6X (consumer card; the official page states no bandwidth figure).
- Apple MacBook Pro (M5 family): unified memory starting at 16/24/32GB,
up to 128GB on higher configs — CPU and GPU share the same memory (official spec page).

Honest note: AMD's consumer-card spec pages could not be verified this pass (site unreachable),
so this site records no AMD figures — unknown stays unknown, no copying from second-hand spec sheets.
Power draw, interconnect, FP8 support and other details follow the official pages and are not expanded here.

生成阶段每产出一个 token，都要把模型权重完整读一遍。
单请求 decode 的每秒 token 数上限 ≈ 显存带宽 ÷ 每 token 需读的字节数。

一个 8B 模型的 FP16 权重约 16GB。在 3.35TB/s 带宽上，
理论上限约 200 token/秒——这是「上限」，实际还要扣 KV 缓存读取与开销。
量化到 4bit 后权重约 4GB，同一张卡上限约四倍。这就是量化的吞吐账。

预填充（读入长提示）相反：一次前向处理多个 token，算力受限。
所以「首 token 延迟」与「每 token 延迟」由两种硬件属性分别决定，
调优方向完全不同。

Every token the generation phase produces requires reading the model weights in full once more.
The per-second token ceiling of single-request decode ≈ memory bandwidth ÷ bytes read per token.

An 8B model's FP16 weights are about 16GB. On 3.35TB/s bandwidth the
theoretical ceiling is roughly 200 tokens/second — that is the ceiling; real numbers also subtract KV-cache reads and overhead.
Quantised to 4-bit the weights are about 4GB, so the same card's ceiling is about four times higher. That is the quantisation throughput ledger.

Prefill (reading a long prompt) is the opposite: one forward pass processes many tokens, compute-bound.
So “time to first token” and “time per token” are set by two different hardware properties,
and the tuning directions are completely different.

线上推理成本的构成链是：

• 每 token 成本 = 硬件每小时成本 ÷ 该硬件每小时产出的 token 数。
• 每小时产出由上节的带宽账与并发批处理决定——批越大，权重读取摊薄越多。
• 硬件每小时价格随市场每月波动，本站按纪律不收录。

能自己算的是中间项。给定模型大小与量化档，再给定并发数，
用带宽账估吞吐，用吞吐除单价。厂商报价页与云厂商价格页是公开入口，
本站只教这条链怎么搭，不替你抄当下的数。

选型判断依据（不是推荐）：并发高、上下文长时，瓶颈在 KV 缓存显存，要看带宽与容量。
单机个人用则相反，瓶颈在「装不装得下」，主要看容量与量化档。

The cost chain for online inference decomposes as:

• Cost per token = hardware cost per hour ÷ tokens that hardware produces per hour.
• Hourly output is set by the bandwidth math above plus concurrent batching — the bigger the batch, the more the weight read is amortised.
• Hardware price per hour moves with the market monthly; this site does not store it, by discipline.

What you can compute yourself is the middle term. Given model size and quantisation tier, plus a concurrency number,
estimate throughput with the bandwidth math, then divide throughput into the unit price. Vendor quote pages and cloud price pages are public entry points;
this site teaches how to build the chain, it does not copy today's numbers for you.

Judgement basis for selection (not a recommendation): with high concurrency and long context, the bottleneck is KV-cache memory — watch bandwidth and capacity.
For single-machine personal use it is the opposite: the bottleneck is “does it fit” — mainly capacity and quantisation tier.

本站系列页的「显存倒推」表按实测文件体积给出「你的显存能下哪一档」，
是使用者入口；这一篇给的是硬件侧事实与换算方法，是从业者口径。
两边都将权重占用与 KV 缓存计入内存预算。

The “VRAM reverse lookup” table on this site's series pages answers “which tier your VRAM can download” from measured file sizes —
that is the user's entry point; this guide gives hardware-side facts and the conversion method, the practitioner's yardstick.
Both include weight storage and KV cache in the memory budget.

学 AI 站的 显存 与
量化 是使用者视角的一页科普；
这一篇给硬件事实表与成本数学的构成方法。

同一个概念在两处深度不同，互链不重复。
KV 缓存与批处理机制见上一篇「跑起来深水」。

The Learn site's VRAM and
quantisation pages are user-perspective explainers;
this guide gives the hardware fact table and the composition method for the cost math.

Same concept, different depth on each side, cross-linked without duplication.
For the KV cache and batching mechanics see the previous guide, “Serving deep dive”.

下面是假设的容量练习，不对应具体模型，也不是本站运行结果。GiB 按 2 的 30 次方字节计算；不要把十进制 GB 直接当成相同数值的 GiB。

设可用于任务的预算为 10 GiB，候选权重占 6 GiB。先按 2 GiB 缓存、1 GiB 运行时占用估算，合计 9 GiB，另留 1 GiB 余量。这些缓存与运行时数字只是练习假设；真实数值随模型、精度、上下文和并发变化。

接着核对文件体积、运行器支持和实际可用内存。用短输入启动后记录占用，再增加上下文。若缓存超过预算，可以缩短输入、调整并发或评估其他档位；卸载能力需查运行器文档。

容量预算不能预测生成速度，也不能证明质量足够。对比方案时，再用相同任务记录耗时、错误和用量；本例不给出性能承诺或硬件购买建议。

接着看 本地运行完整流程，或回到 量化概念 核对位宽与质量的区别。

This hypothetical capacity exercise is not a named model or a measured site result. A GiB is 2^30 bytes; decimal GB and GiB are not numerically interchangeable.

Assume a 10 GiB available budget and 6 GiB of weights. With assumed cache use of 2 GiB and runtime use of 1 GiB, the estimate is 9 GiB, leaving 1 GiB headroom. Cache/runtime figures are exercise assumptions; actual use depends on model, precision, context and concurrency.

Check file size, runtime support and available memory. Start with short input, record use, then increase context. If cache exceeds budget, assess shorter input, different concurrency or other tiers. Check runtime documentation for offload support.

Capacity does not predict speed or establish quality. Compare equal tasks for latency, errors and usage. This example makes no performance guarantee or purchase recommendation.

Continue to the local-model workflow, or revisit quantization to separate bit width from quality.