GGUF Quantization: How to Choose Q4 vs Q8 for Local LLMs

When you look for a model to run on your own computer, you will see the same model name followed by suffixes like Q4_K_M, Q5_K_M and Q8_0. These mark the level of GGUF quantization, the technique that lowers the numerical precision of the values inside an AI model so the file gets smaller. If you are unsure, pick the largest model that fits on your PC and use the Q4_K_M quantization.

Quantization does not delete knowledge; it only rounds off numbers, much like JPEG compression for images. In a January 2026 arXiv study of Llama 3.1 8B, Q4_K_M cut the file from 15.3GB to 4.6GB and lowered the MMLU knowledge-test score by only about 1.1 points. Q8_0 is almost lossless, but its file is about twice as large and generation is roughly 29% slower.

The same model at a different quantization level is effectively a different product, so it helps to know how to read the labels and what each level costs.

---

What is quantization and how does it shrink local LLMs?#

Quantization reduces the number of bits used for each numerical value inside an AI model. It does not erase knowledge; it only makes the values less precise. In the baseline FP16 format each value uses 16 bits, so a Q4 version is roughly one-fourth the size, which brings models of tens of gigabytes down to sizes a home PC can hold.

An AI model is a huge collection of numbers called weights. They are set during training and work as the model's stored memory. FP16 uses 16 bits (2 bytes) per weight, while Q8 uses about 8 bits and Q4 about 4.

A helpful analogy is JPEG compression. The subject of a compressed photo is still recognizable, and you only notice the roughness when you zoom in. A quantized model behaves the same way: the knowledge stays, and the numbers become coarser. That is the trade-off, a smaller file in exchange for slightly sloppier detail.

The effect shows up more in smaller models. At around 1B parameters, quantizing to Q4 can badly degrade response quality, so smaller models are better kept at Q5 or Q6.

What does Q4_K_M mean in GGUF filenames?#

Q4_K_M splits into four parts: Q for quantization, 4 for the approximate bit count, K for the K-quant method, and M for the mixing strength. Knowing this lets you guess a file's quality and size from its name. Q4_K_M is the usual default balance between file size and quality.

GGUF is the model file format used by local runtimes such as llama.cpp and LM Studio. On Hugging Face, the same model appears as many files such as Q4_K_M, Q5_K_M, Q6_K, Q8_0 and IQ4_XS. Each one is the same model quantized at a different level.

• Q (quantization): marks the file as quantized.
• 4 (bit count): the approximate number of bits per value. A larger number means higher quality and a heavier file.
• K (K-quant): a mixed method introduced to llama.cpp in 2023 that varies precision by layer.
• M (mixing strength): S is small, M is medium and L is large.

K-quants give more bits to layers that are sensitive to precision, such as the attention layers, and compress less sensitive layers more aggressively. At the same size this beats the older Q4_0. Unless you have a specific reason, _M is the usual choice.

IQ variants such as IQ4_XS use an importance matrix (imatrix), a measurement made beforehand, to give precision to the weights that matter most. At low bit counts they can beat K-quants, but files made without an imatrix end up worse. IQ4_XS runs at about 4.25 bits per weight (bpw) and is reported to match or exceed Q4_K_M in quality at a slightly smaller size.

Check who converted the file as well as the quantization level. The same base model processed by a different builder is effectively a different model (see what abliteration is).

Which quantization level should you choose between Q4 and Q8?#

Use Q4_K_M as the default, and step up to Q6_K if you have spare VRAM. In the Llama 3.1 8B measurements, Q4_K_M lost only 1.1 MMLU points against uncompressed FP16 while shrinking the file from 15.3GB to 4.6GB. Q8_0 is near-lossless, but its file is about twice the size and generation is roughly 29% slower.

A January 2026 paper on arXiv measured Llama 3.1 8B at each quantization level. It reports perplexity (how uncertain the model is about the next word; lower is better), MMLU accuracy (a knowledge test) and file size:

• FP16 (uncompressed baseline): perplexity 7.32 | MMLU 63.5% | 15.3GB
• Q6_K: perplexity 7.35 | MMLU 63.2% | 6.3GB
• Q5_K_M: perplexity 7.40 | MMLU 62.8% | 5.5GB
• Q4_K_M: perplexity 7.56 | MMLU 62.4% | 4.6GB

Three things follow from these numbers. Q4_K_M cuts 15.3GB to 4.6GB at a cost of about 1.1 MMLU points, and perplexity worsens by about 3%. For everyday conversation, summarization and lookups, that difference is unlikely to be noticeable.

Q6_K is a little over 40% of the FP16 size and nearly equal in quality (perplexity +0.03). Q8_0 comes even closer to lossless (+0.01), but its file is about twice that of Q4_K_M and generation is about 29% slower, because memory bandwidth becomes the bottleneck. When you only feed in long text, as in summarization or search, the speed difference is hard to notice, and near-lossless Q8_0 works well.

The quality cliff lies between Q3 and Q4. Below Q4, Q3 and Q2 need an imatrix-based IQ variant to hold up, and Q2_K suits only models of 30B parameters or more. This is why Q4 is treated as the default.

How do you choose the right quantization for your PC's VRAM?#

Estimate the requirement as the quantized model size plus 10 to 20% headroom for context, the text you feed in at once. Start by finding the largest model that fits at Q4_K_M. At the same VRAM, a larger model at Q4 usually beats a smaller model at Q8, so prefer model size over quantization precision.

Typical VRAM requirements by model size, from left to right for Q4_K_M, Q5_K_M and Q8_0:

• 3B: ~2.5GB → ~3GB → ~4GB
• 7B to 8B: ~5–6GB → ~6–7GB → ~9–10GB
• 13B: ~8–9GB → ~10–11GB → ~15–16GB
• 30B to 34B: ~19–22GB → ~23–26GB → ~34–38GB
• 70B: ~38–42GB → ~47–51GB → ~72–76GB

To check your own numbers, put the model size, context length and memory into the GGUF VRAM calculator. It returns a comfortable / tight / too-big verdict for each quantization, and publishes the formula including the KV cache and reserve.

Follow these four steps:

1. Check your PC's memory or GPU memory. The procedure is in our local LLM beginner's guide.
2. Pick the largest size that fits at Q4_K_M using the list above.
3. If it still does not fit, drop one quantization level or move to a smaller model.
4. If you have headroom, step up to Q5_K_M or Q6_K.

As an example, take a 27B uncensored model. The abliterated variants of Qwen3.8-27B have no official GGUF, so every quantization is a community conversion. One distribution measures 15.41GiB at Q4_K_M, 11.24GiB at Q3_K_S and 26.63GiB at Q8_0. A 27B model therefore needs 16GB or more even at Q4, which is heavy for a first model. We compare the options in our roundup of the best uncensored models.

For a first model, a 7B-class model on a PC with 8 to 16GB of memory is the realistic choice.

What is dynamic quantization and how does it allocate bits?#

Dynamic quantization decides which layers to keep precise and which to compress by measuring each model, instead of applying a fixed recipe. Unsloth Dynamic 2.0 is the best-known example, and you will see it in filenames such as UD-Q4_K_XL. Its claim of Q5-like quality at Q4 size is a vendor figure, not yet independently verified.

Traditional quantization applies roughly the same bit allocation to each type of layer. Dynamic quantization measures each model and layer and distributes the bits accordingly. It works on Mixture-of-Experts (MoE) models, which split the network into specialist parts, as well as on standard models, and it uses calibration data of more than 1.5 million tokens to decide the allocation.

The published figure is "Q5-like quality at Q4 size." It is reported by the vendor, and third-party verification is still in progress.

The idea extends the imatrix and IQ approach of measuring which weights matter and allocating precision to them. Allocation is moving from a fixed static recipe to one based on measurement.

What should you check before using quantized models?#

What is Deepent Research?#

Deepent Research is a browser-accessible LLM chat service built to refuse as little as possible. It keeps going with research that a standard AI would cut off mid-answer.

• It refuses nothing. We run uncensored and abliterated models with guardrails reduced as far as possible, so prompts a standard AI answers with "I can't help with that" still get a response (Qwen3.5 / Qwen3.8-27B-abliterated, depending on your plan).
• Your input is never used for training. What you look up is used only to provide the service, never to train AI or for any other purpose. Personal data is protected, and our privacy policy is public.
• Try it free. No GPU and no install, just open a browser. In the logged-out trial your conversation isn't saved, so you can start casually.

→ Try it free, no sign-up: https://deepentresearch.com

No question can stop research.

Frequently asked questions#

Does quantization make an AI model significantly less capable?#

⌄

No. Quantization does not erase a model's knowledge; it only makes the stored numbers less precise. In a January 2026 test on Llama 3.1 8B, Q4_K_M lowered the MMLU knowledge-test score by only about 1.1 points compared with uncompressed FP16. For everyday conversation, summarization and lookups, that difference is unlikely to be noticeable.

Should you choose Q4 or Q8 for GGUF models?#

⌄

If you are unsure, Q4_K_M is the default. Q8_0 is almost lossless, but in published tests its file was about twice as large and generation was about 29% slower, because memory bandwidth becomes the bottleneck. For tasks that mostly read long text, such as summarization or search, the speed difference is small, so Q8_0 is worth considering there.

Is it safe to quantize smaller models down to Q4?#

⌄

Smaller models show the effect of quantization more. At around 1B parameters, quantizing to Q4 can badly degrade response quality. For smaller models, choose a level closer to Q5 or Q6 instead, which keeps more precision at the price of a somewhat larger file.

Can you use GGUF quantization levels below Q4 such as Q3 or Q2?#

⌄

The quality cliff lies between Q3 and Q4. Below Q4, a file generally needs to be an imatrix-based IQ variant to hold up, and Q2_K suits only large models of 30B parameters or more. For most uses, stay at Q4 or above.

Disclaimer#

This article was drafted with AI assistance and then reviewed, fact-checked, and edited by the Deepent Research team before publication.
It is provided for educational and research purposes and is not intended to encourage unlawful activity or crime.
Where figures are cited, they come from third-party public reports and vendor-published values and may vary with measurement conditions.