Why this matters
- A correct PolicyDesk that takes 20 seconds and a full GPU for one user will not be deployed.
- Interviewers expect the menu: quantization, KV cache, batching, an optimized runtime, parallelism, speculative decoding, streaming. They also expect you to say which symptom each one treats.
- These techniques stack. They also interfere. Quantizing a cache and a weight file are related but not the same knob.
- "We used vLLM" is not an explanation until you say which problems vLLM is solving.
The menu
| Technique | What it changes | The trade |
|---|---|---|
| Quantization (INT8, INT4, FP8) | Stores weights, and sometimes activations, in fewer bits | Smaller memory and faster math. Some quality loss, worse if you quantize aggressively without calibration. |
| KV cache | Keeps past key and value tensors so the next token does not recompute them | Much faster decoding. The cache grows with sequence length and batch size, and it can dwarf the weights. |
| Batching | Runs several requests through the GPU together | Higher throughput. A slow request can hold up the batch if the batch is static. |
| Dynamic or continuous batching | Adds and removes requests from the running batch as they arrive and finish | Better GPU utilization under uneven prompt lengths. More scheduler complexity. vLLM is known for this. |
| Optimized runtimes | vLLM, TensorRT-LLM, ONNX Runtime, TGI | Fused kernels, paged KV cache, better memory planning. You operate a serving stack, not a Python for-loop. |
| Sharding and parallelism | Tensor or pipeline parallelism splits a model across GPUs | Lets you serve a model that does not fit on one device. Communication between GPUs becomes part of the latency. |
| Speculative decoding | A small draft model proposes several tokens; the large model checks them in one pass | Lower latency when the draft is often right. Wasted work when the draft is often wrong. You now run two models. |
| Streaming | Sends tokens to the client as they are generated | Better experience. Does not reduce the work. Still needs a timeout and a max token cap. |
Quantization (INT8, INT4, FP8)
What it changesStores weights, and sometimes activations, in fewer bitsThe tradeSmaller memory and faster math. Some quality loss, worse if you quantize aggressively without calibration.KV cache
What it changesKeeps past key and value tensors so the next token does not recompute themThe tradeMuch faster decoding. The cache grows with sequence length and batch size, and it can dwarf the weights.Batching
What it changesRuns several requests through the GPU togetherThe tradeHigher throughput. A slow request can hold up the batch if the batch is static.Dynamic or continuous batching
What it changesAdds and removes requests from the running batch as they arrive and finishThe tradeBetter GPU utilization under uneven prompt lengths. More scheduler complexity. vLLM is known for this.Optimized runtimes
What it changesvLLM, TensorRT-LLM, ONNX Runtime, TGIThe tradeFused kernels, paged KV cache, better memory planning. You operate a serving stack, not a Python for-loop.Sharding and parallelism
What it changesTensor or pipeline parallelism splits a model across GPUsThe tradeLets you serve a model that does not fit on one device. Communication between GPUs becomes part of the latency.Speculative decoding
What it changesA small draft model proposes several tokens; the large model checks them in one passThe tradeLower latency when the draft is often right. Wasted work when the draft is often wrong. You now run two models.Streaming
What it changesSends tokens to the client as they are generatedThe tradeBetter experience. Does not reduce the work. Still needs a timeout and a max token cap.
Why you reach for them
The symptoms
- Latency. Time to first token and time to finished answer. Speculative decoding, a smaller quantized model, and a warm KV cache for multi-turn chats target latency. Streaming hides some of it.
- Throughput. Requests per second and tokens per second across users. Batching and continuous batching exist for this. A batch of one is a GPU running mostly idle.
- Memory and money. Quantization and paged KV caches let you serve more concurrent sequences on the same card, or serve a larger model on a smaller card.
- Fit. A model that does not fit is not slow. It does not run. Sharding or a quantized checkpoint is what makes it exist on your hardware.
- Quality. Every technique above can dent accuracy, especially aggressive quantization and a sloppy draft model. You measure faithfulness on the PolicyDesk set before you keep the speedup.
People quote improvements like "tens of percent faster" as if they were a property of the technique. They are a property of the baseline. A naive loop that recomputes the whole prompt every token will look miraculous after a KV cache. A service already on vLLM will not gain another 70 percent from a blog post. Measure your baseline. Say the number you measured.
The rest of this guide is part of Pro
Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.
Still to come · 2 sections
- Where they sit in the path
- What you should claim as a result
Test yourself
Answer these before moving on — recall is what makes it stick.