Why this matters
- PolicyDesk does not need a giant general model to quote an 8-day leave rule, but it does need the quote to stay accurate.
- Interviews ask for the difference, the quality risk, and when you would combine them.
- Picking INT4 because a chart said "4x smaller" without an eval set is how a faithful assistant becomes a confident one.
- These are training-time and checkpoint-time decisions. They are not prompt tricks.
Two different moves
Distillation. A large teacher produces soft targets: a full probability distribution over the next token, not only the chosen word. A smaller student trains to match those distributions (often with a temperature that softens them) and sometimes the hard label as well. The student can also be nudged to match intermediate hidden states. You end with a different, smaller network that hopefully behaves like the teacher on the tasks you distilled.
Quantization. You take a trained model and represent weights, and sometimes activations, in lower precision. FP32 to FP16 or BF16 is the ordinary serving step and is usually nearly free in quality. INT8 and INT4 are the aggressive steps people mean by "we quantized." Each weight is stored as a small integer plus a scale (and sometimes a zero-point) that maps it back to a real number at multiply time.
| Aspect | Distillation | Quantization |
|---|---|---|
| What changes | The architecture. Fewer layers or smaller hidden size. | The numeric format of an existing architecture. |
| When | A training job. You need teacher outputs or a dataset. | After training (post-training quantization) or during training (quantization-aware training). |
| What you need | A teacher, compute, and a task corpus. | A calibration set for good PTQ, or training data for QAT. Not a teacher. |
| Typical win | A model that is cheaper every day because it has fewer FLOPs. | A model that fits in less memory and uses faster integer or low-precision kernels. |
What changes
DistillationThe architecture. Fewer layers or smaller hidden size.QuantizationThe numeric format of an existing architecture.When
DistillationA training job. You need teacher outputs or a dataset.QuantizationAfter training (post-training quantization) or during training (quantization-aware training).What you need
DistillationA teacher, compute, and a task corpus.QuantizationA calibration set for good PTQ, or training data for QAT. Not a teacher.Typical win
DistillationA model that is cheaper every day because it has fewer FLOPs.QuantizationA model that fits in less memory and uses faster integer or low-precision kernels.
You can distill and then quantize the student. They solve different constraints.
How distillation works
Soft targets carry "the teacher thought these three phrasings were all plausible." A hard label throws that away. Matching the soft targets is why a student can inherit some of the teacher's judgment without copying every weight. Temperature in the softmax (a number above 1) flattens the distribution so the student sees more than the single peak.
Task-specific distillation is the practical version for PolicyDesk: distill on grounded question-answer pairs from the handbook, not on the entire internet. The student then does not need to be a general genius. It needs to follow context and quote numbers. It may still lose multi-step reasoning the teacher had. Test reasoning separately if you care about it.
The rest of this guide is part of Pro
Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.
Still to come · 3 sections
- How quantization works
- What you gain and what you lose
- When to use which
Test yourself
Answer these before moving on — recall is what makes it stick.