# Quantization

> Quantization stores a model’s weights at lower numerical precision — 4-bit or 8-bit integers instead of 16-bit floats — so the model needs less memory and…

Source: https://northerngo.com/glossary/quantization/
Language: en
Updated: 2026-08-28

---
**Quantization stores a model’s weights at lower numerical precision — 4-bit or 8-bit integers instead of 16-bit floats — so the model needs less memory and runs faster. It is what makes it possible to run a multi-billion-parameter model on a consumer graphics card.**

### The arithmetic that decides what fits

Memory for weights is roughly parameters times bytes per parameter. An 8-billion-parameter model needs about 16 GB at 16-bit precision, about 8 GB at 8-bit and about 4 to 5 GB at 4-bit. That is the difference between "needs a datacentre card" and "runs on a laptop", and it is the entire reason quantization is everywhere in [local AI](/glossary/local-ai/).

Speed improves too, and for a reason that is easy to miss: generating tokens is limited by how fast weights can be read from memory, not by arithmetic. Halving the bytes roughly halves the reading.

You will meet quantization mostly as filename suffixes. `Q4_K_M` and `Q8_0` are GGUF variants used by llama.cpp and [Ollama](/glossary/ollama/); `q4f16_1` is the MLC convention used by browser runtimes on [WebGPU](/glossary/webgpu/). The important part is the bit width and whether it is a mixed scheme that keeps sensitive layers at higher precision — most good 4-bit formats do.

### What to watch out for

**Quantization does not make the model smaller in the sense of removing knowledge — it makes each number less precise.** The parameter count is unchanged. That distinction matters because it predicts where the damage shows up: not as forgotten facts but as slightly worse judgement, less reliable formatting and more drift on long outputs.

Three things worth knowing:

- **The quality curve is not linear.** 8-bit is usually indistinguishable from full precision. 4-bit is a mild, often acceptable loss. Below 4-bit, degradation becomes obvious fast.
- **A bigger model quantized harder usually beats a smaller model at full precision**, for the same memory budget. This is the standard advice and it holds surprisingly well.
- **Losses are task-dependent.** Chat and summarising survive 4-bit well. Precise instruction-following, structured output and long chains of reasoning are hit hardest — exactly the things code generation depends on. If a quantized model keeps breaking a required output format, that is a plausible cause.

Test on your own workload rather than trusting a perplexity number. Perplexity barely moves while the behaviour you care about degrades.

---

NorthernGo is an AI-powered platform for building production-ready web apps with zero coding. Local AI generation via WebGPU is unlimited and free, and you own all generated source code. https://northerngo.com/