Quantization
What is it?
Quantization shrinks an AI model by storing its numbers with less precision, making it smaller and faster with usually only a small quality loss.
Explain like I'm 5
Why was it created?
Large models are heavy and slow. Quantization was adopted to run them on cheaper hardware and with lower latency.
Where is it used?
- Running models on limited hardware
- On-device and edge AI
- Reducing inference cost
- Faster model serving
Why should developers care?
It's a key technique for deploying models efficiently, including on phones and modest servers, so it matters for practical AI.
How does it work?
A model's parameters are normally stored as high-precision numbers. Quantization converts them to lower-precision formats (using fewer bits), cutting memory and speeding up math, with techniques to limit accuracy loss.
Real-world example
A quantized version of a language model runs on a laptop or phone that couldn't fit the full-precision original.
Common use cases
- On-device inference
- Cheaper, faster serving
- Fitting bigger models in limited memory
- Edge deployments
Advantages
- Smaller memory footprint
- Faster inference
- Lower hardware cost
- Enables on-device AI
Disadvantages
- Some accuracy loss
- Not all models quantize equally well
- Requires care to do well
- Tooling/hardware support varies
When should you use it?
When you need a model to be smaller, cheaper, or faster and can accept a small quality trade-off.
When should you avoid it?
When maximum accuracy is critical and you have the hardware for full precision.
Alternatives
Related terms
Interview questions
Beginner
- What does quantization do?
- Why make a model smaller?
Intermediate
- What is the trade-off of quantization?
- Why does lower precision speed up inference?
Senior
- When is accuracy loss from quantization acceptable?
- How does quantization enable on-device models?
Common misconceptions
- "Quantization always wrecks accuracy" — done well, the quality drop is often small.
- "Quantization changes what the model knows" — it reduces numeric precision, not the model's learned knowledge.
Fun facts
- Lower-precision number formats can dramatically cut a model's memory use.
- Quantization is a big reason capable models can now run on phones.
Timeline
- 2020s — Quantization becomes key to running large models efficiently
Learning resources
Quick summary
Quantization stores a model's numbers at lower precision to make it smaller and faster, usually with only minor accuracy loss.
Cheat sheet
- Lower-precision model weights
- Smaller + faster
- Small accuracy trade-off
- Enables on-device AI