多模态模型的显存账怎么算Counting memory for a multimodal model从业者视角的方法论 · 关键事实挂官方出处practitioner-perspective methodology · key facts carry official sources

多模态的显存账是三笔账的合计，不是一个数字。

三笔是：视觉或音频编码器的占用、投影件与接入层的占用、解码器侧的权重与缓存。

本站不跑模型、不读硬件，也不给某个多模态模型的占用结论。页面能给的只有分法、单位和边界。

Memory for a multimodal model is the total of three separate bills, not one number.

The three are: encoder occupancy, projector and connector files, and the decoder side's weights plus cache.

This site runs no models, reads no hardware and issues no occupancy conclusion for any multimodal model. What it can give is the split, the units and the boundaries.

只问「要多大显存」问不出答案，先按层拆开，每笔单独记来源。

• 文本权重：解码器主干驻留的体积。它是驻留估计，不等于下载文件字节数。
• 编码器与投影件：图像、音频通路自己的权重，运行器把它们作为单独文件加载。
• 运行时与临时张量：预处理、批调度、CUDA Graph 一类开销，官方未列明时留空。
• 解码器缓存：图像变成 token 进入上下文之后才产生的那一笔 KV。

三笔里任何一笔用错单位，合计就没有意义。本页一律按 GiB 记账，十进制 GB 先换算。

Asking "how much VRAM does it need" has no answer until the layers are split and each line records its own source.

• Text weights: the resident volume of the decoder backbone. A residency estimate, not the downloaded file's byte count.
• Encoder and projector: the image/audio path's own weights, which the runner loads as separate files.
• Runtime and temporary tensors: preprocessing, scheduling and CUDA-graph style overhead; left blank when not documented.
• Decoder cache: the KV that appears only after image tokens enter the context.

If one line uses the wrong unit, the total is meaningless. This page books everything in GiB and converts decimal GB first.

多模态不是一句「模型支持图片」，它有自己的文件与开关。

• 运行器侧要同时给出文本文件与投影件：官方文档写的是 -m model.gguf 配 --mmproj file.gguf，各指一端。出处见 docs/multimodal.md。
• 投影件默认卸载到 GPU，官方文档给了关掉它的开关 --no-mmproj-offload。这直接影响它算在显存还是内存里。
• 投影件不止投影：官方 quantize 文档说明这个文件里可能包含视觉或音频编码器等多个部件。
• 转换与量化要分别做：官方文档写明支持图像或音频输入的模型，其编码器与投影件需要单独转换和量化。
• OCR 一类模型对提示与输入结构有自己要求，官方文档提醒按模型说明处理，不能只按聊天用法喂图。

官方 quantize 文档另有一句精度口径：这些部件通常用较高精度保存。用更小的量化档，对速度与内存影响不明显，但可能影响整体质量。它是官方口径，不是本站实测。

Multimodal support is not the sentence "it accepts images"; it has its own files and switches.

• The runner needs the text file and the projector file at once: the official documentation describes -m model.gguf together with --mmproj file.gguf, one per side. Source: docs/multimodal.md.
• The projector is offloaded to the GPU by default, and the same documentation gives --no-mmproj-offload to stop that. This decides whether the line counts as VRAM or system memory.
• A projector file is more than a projection: the official quantize documentation says it may contain vision or audio encoders and other components.
• Conversion and quantisation are done separately: the documentation states that models accepting image or audio input need their encoders and projectors converted and quantised as well.
• OCR-style models carry their own prompt and input-structure requirements; the documentation points back to the model's instructions rather than chat-style feeding.

The official quantize documentation adds a precision note: these components are usually kept in a high-quality format, and using a smaller quant affects speed and memory only slightly but can affect overall quality. That is the vendor's statement, not a measurement by this site.

图像进入解码器时变成多少 token，由模型与它的预处理决定。这不是能从体积推出来的数。

• 该数值要从官方模型配置、预处理说明或运行器文档取。三处都取不到时，本站写官方未说明，不用张数猜测。
• 同一模型换分辨率、换缩放策略，token 数会变，缓存那一笔跟着变。
• 每次请求能带几个图像或几段视频，是可以限制的设置项。vLLM 的内存控制文档给了 limit_mm_per_prompt 这类参数，见 conserving_memory。
• 官方文档也提醒某些模型需要更大的上下文窗口，例如把上下文设到 8192 的示例。上下文变大主要是缓存那一笔在涨。

How many tokens an image becomes inside the decoder is set by the model and its preprocessing. It cannot be derived from file size.

• Take the figure from the official model configuration, preprocessing description or runner documentation. When none of the three gives it, this site writes that it is not stated rather than guessing from patch counts.
• The same model at another resolution, or with another resizing strategy, yields a different token count, and the cache line moves with it.
• How many images or video clips a request may carry is a limitable setting. vLLM's memory documentation lists parameters such as limit_mm_per_prompt; see conserving_memory.
• The official documentation also warns that some models need a larger context window, with an example setting the context to 8192. A longer context mainly grows the cache line.

解码器缓存的标准算法是 2 × L × Hkv × D × B × C × N。字节结果除以 1,073,741,824 才是 GiB。

它成立的口径很窄。

适用对象是标准全注意力 MHA / GQA 解码器。前提还有两条：每层保留完整键值，各层结构相同。

MLA、混合状态空间、滑动窗口与逐层缓存不同的架构都不适用。编码器侧根本不产生这种缓存。

• C 要按「提示 token 加图像转出的 token 再加已生成内容」计，不是只算文字长度。
• N 假设各请求长度相同、缓存互不共享；分页复用与共享前缀会改变口径。
• vLLM 的 PagedAttention 设计文档描述的是按块存放每头固定 token 数的缓存，见 paged_attention。
• 缓存精度是独立设置项，官方把缓存空间以 GiB 配置的例子见同一份内存控制文档。

字段与算式在容量工作台上逐项填写：上下文与并发预算。

The standard decoder cache calculation is 2 × L × Hkv × D × B × C × N; divide the byte result by 1,073,741,824 to obtain GiB.

Its scope is narrow: standard full-attention MHA/GQA decoders with identical layers and complete key/value state kept at every layer.

MLA, hybrid state-space designs, sliding windows and architectures whose caches differ per layer are outside it, and encoders never produce this kind of cache.

• Count C as prompt tokens plus tokens produced from the image plus generated content, not text length alone.
• N assumes equal request lengths and no cache sharing; paging reuse and shared prefixes change the basis.
• vLLM's PagedAttention design document describes blocks holding a fixed number of tokens per head; see paged_attention.
• Cache precision is a separate setting; the same memory documentation configures cache space in GiB.

Fill the fields and see the arithmetic in the capacity workbench: context and concurrency budget.

本节数字全是虚构教学假设，不对应任何真实模型、官方页或本站运行结果。

设文本权重按 5 GiB 计，编码器与投影件合计按 1.5 GiB 计。运行时与预留按 1 GiB 计。

再假设每张图转 256 token，一次请求带 2 张图。提示与生成内容合计 1,500 token，因此 C 按 2,010 计。并发 N 取 1。

按同层标准全注意力 MHA / GQA、L 32、Hkv 8、D 128、B 为 2 字节练习：KV 约 0.2 GiB，合计约 7.7 GiB。

• 若架构确认不了（MLA、混合或滑动窗口），这条算式作废，只能回官方方法另算。
• 若任务可用内存按 8 GiB 计，假设下留 0.3 GiB 余量。这个数字不承诺跑得起来，也不承诺图像理解的质量。
• 想缩短结果，可减图像数量、降上下文长度或改并发，每一项都会改变上面那笔缓存。

Every number in this section is a fictional teaching assumption. It matches no real model, official page or result produced by this site.

Assume text weights of 5 GiB, an encoder plus projector total of 1.5 GiB, and 1 GiB for runtime and reserve.

Now assume each image becomes 256 tokens, one request carries 2 images, and prompt plus generated content totals 1,500 tokens, so C is 2,010 with N = 1.

Under a standard same-layer full-attention MHA/GQA assumption with L 32, Hkv 8, D 128 and B at 2 bytes, KV is about 0.2 GiB and the total about 7.7 GiB.

• If the architecture cannot be confirmed (MLA, hybrid or sliding window), this calculation is void and the official method for that architecture must be used instead.
• Against an 8 GiB task-available budget the assumption leaves 0.3 GiB headroom. That figure promises neither a successful start nor image-understanding quality.
• To shrink the result, reduce image count, context length or concurrency. Each change moves that cache line.

• 载入失败先查运行器是否支持该架构与投影件组合，按症状进入 故障库。
• 占用超限的排查入口是 容量不足排查，先确认单位与假设再改设置。
• 结果判定走 20 项自测清单，图像任务要写清可核对的字段。
• 任务接口层面的差别（OCR、语音、嵌入、重排）见本站「任务模型」专题，那边不给解码器 KV 公式套编码器。
• 要按任务记录费用构成时用 成本计算器，本站不保存单价，也不预测某台机器的产出。

• Load failures: first check whether the runner supports this architecture and projector combination; start at the failure library.
• Over budget: the entry point is memory troubleshooting. Confirm units and assumptions before changing settings.
• Acceptance: use the 20-item self-test and write checkable fields for image tasks.
• For interface-level differences (OCR, speech, embedding, reranking) see the "Task models" guide on this site; the decoder KV formula is not applied to encoders there either.
• To record cost components per task use the cost calculator; this site stores no unit prices and predicts no machine output.

学 AI 站的 多模态、编码器与解码器 与 显存 是给使用者的一页科普。

这一篇把占用拆成三笔账，并写明每笔的来源与算式适用范围。

同一个概念在两处深度不同，互链不重复。带宽与成本构成见「GPU 事实与成本数学」。

The Learn site's multimodal, encoder and decoder and VRAM pages are one-page explainers for users.

This guide splits occupancy into three bills and states the source and scope for each.

Same concept, different depth on each side, cross-linked without duplication. For bandwidth and cost composition see "GPU facts and cost math".