TOOL
GGUF VRAM calculator
Running a quantized model takes the weights, a KV cache for your context, and a runtime reserve. This page adds all three. Enter the model size, the context length you want and your memory, and it tells you which quantizations are comfortable, tight, or too big.
e.g. 8, 12, 27, 32, 70
Advanced (defaults are fine)
KV cache scales with layers
CUDA context, display, fragmentation
Memory by quantization
Recommended: IQ4_XS （good for daily use / 6.43 GB）
Comfortable means the total stays under 85% of your memory; the rest goes to the OS and display.
| Quant | Weights | KV cache | Total | Verdict |
|---|---|---|---|---|
| Q8_0 | 8.5 GB | 1.13 GB | 10.63 GB | Too big |
| Q6_K | 6.6 GB | 1.13 GB | 8.73 GB | Too big |
| Q5_K_M | 5.7 GB | 1.13 GB | 7.83 GB | Tight |
| Q4_K_M | 4.9 GB | 1.13 GB | 7.03 GB | Tight |
| IQ4_XS← | 4.3 GB | 1.13 GB | 6.43 GB | Comfortable |
| Q3_K_M | 4 GB | 1.13 GB | 6.13 GB | Comfortable |
| Q2_K | 3.2 GB | 1.13 GB | 5.33 GB | Comfortable |
Estimates in GB; totals include the reserve.
How to read the verdicts
- Comfortable: the total stays under 85% of your memory, leaving room for long chats and image input.
- Tight: it fits, but with little left over. Halving the context length usually makes this stable.
- Too big: it does not fit. Step the model size down or shorten the context.
The 85% line exists because GPU memory is never only weights. The CUDA context, the display and fragmentation take 1–2 GB first. The reserve field covers that, and defaults to 1 GB.
The formula
The calculator runs on these three lines. If a number looks wrong, come back here and check it.
weights(GB) = params(B) × bits per weight ÷ 8
kv(GB) = 2 × layers × kv_dim × bytes_per_value × tokens ÷ 1024³
required(GB) = weights + kv + reserve
The 2 in the KV line is for Key and Value. kv_dim is KV heads × head dimension; recent models with grouped-query attention mostly use 1024 (8 heads × 128), which is why it is the default. The exact figure is in the model's config.json.
Bits per weight comes from measured file sizes: for an 8B model, Q8_0 is about 8.5 GB, Q6_K 6.6 GB, Q5_K_M 5.7 GB, Q4_K_M 4.9 GB, IQ4_XS 4.3 GB, Q3_K_M 4.0 GB and Q2_K 3.2 GB. Expect a few percent of variation between publishers and models.
Five things people get wrong
- Counting the weights and forgetting the KV cache, which grows with context length.
- Lowering the quantization before lowering the model size. At a fixed memory budget, a bigger model at a lower quant is usually stronger.
- Reading the box number as usable memory. A 24 GB card gives you roughly 22 GB after the OS and display.
- Entering total unified memory on a Mac. The GPU only gets a share of it.
- Measuring at the maximum context length. Halving it halves the KV cache and can change the verdict.
Frequently asked questions
01Should I pick Q4_K_M or Q8_0?
If it fits, quality falls in this order: Q8_0, Q6_K, Q5_K_M, Q4_K_M. The visible drop only starts below Q4. When in doubt, take Q4_K_M; with headroom, Q5_K_M or Q6_K. With a fixed memory budget, a larger model at a lower quant usually beats a smaller model at a higher one.
02Can I just use a lower quantization on a small PC?
Yes, but Q3_K_M and below start to show visible quality loss. Step the model size down first, and only lower the quantization if that is not enough.
03Is the weight size the same as the VRAM I need?
No. The KV cache for your context is separate and grows with context length: roughly 1–2 GB at 8K and 4–8 GB at 32K, depending on layer count and KV dimension.
04Does this work for Apple unified memory?
The formula is the same, but the GPU only gets part of the total memory — the OS and apps take theirs first. Enter the amount the GPU can actually use, not the total you bought.