Quantization
Quantization stores a model’s weights at lower numerical precision — 4-bit or 8-bit integers instead of 16-bit floats — so the model needs less memory and runs faster. It is what makes it possible to run a multi-billion-parameter model on a consumer graphics card.
The arithmetic that decides what fits
Memory for weights is roughly parameters times bytes per parameter. An 8-billion-parameter model needs about 16 GB at 16-bit precision, about 8 GB at 8-bit and about 4 to 5 GB at 4-bit. That is the difference between "needs a datacentre card" and "runs on a laptop", and it is the entire reason quantization is everywhere in local AI.
Speed improves too, and for a reason that is easy to miss: generating tokens is limited by how fast weights can be read from memory, not by arithmetic. Halving the bytes roughly halves the reading.
You will meet quantization mostly as filename suffixes. Q4_K_M and Q8_0 are GGUF variants used by llama.cpp and Ollama; q4f16_1 is the MLC convention used by browser runtimes on WebGPU. The important part is the bit width and whether it is a mixed scheme that keeps sensitive layers at higher precision — most good 4-bit formats do.
What to watch out for
Quantization does not make the model smaller in the sense of removing knowledge — it makes each number less precise. The parameter count is unchanged. That distinction matters because it predicts where the damage shows up: not as forgotten facts but as slightly worse judgement, less reliable formatting and more drift on long outputs.
Three things worth knowing:
- The quality curve is not linear. 8-bit is usually indistinguishable from full precision. 4-bit is a mild, often acceptable loss. Below 4-bit, degradation becomes obvious fast.
- A bigger model quantized harder usually beats a smaller model at full precision, for the same memory budget. This is the standard advice and it holds surprisingly well.
- Losses are task-dependent. Chat and summarising survive 4-bit well. Precise instruction-following, structured output and long chains of reasoning are hit hardest — exactly the things code generation depends on. If a quantized model keeps breaking a required output format, that is a plausible cause.
Test on your own workload rather than trusting a perplexity number. Perplexity barely moves while the behaviour you care about degrades.
Frequently asked questions
Which quantization level should I pick?
Start at 4-bit in a mixed scheme such as Q4_K_M; it is the usual sweet spot and fits most consumer hardware. Move to 8-bit if you have the memory and the task needs precise formatting or structured output. Go below 4-bit only when nothing else fits, and test before trusting it.
Build it yourself
NorthernGo turns a plain-text description into a working web app with a database, login and a live URL. Local AI generation runs on your own GPU, is unlimited, and is free on every plan.