Why this matters
- PolicyDesk's p95 jumped when one user pasted the whole handbook even though "requests per second" looked calm.
- Interviewers ask you to distinguish batch size, batched tokens, prefill, and decode.
- These numbers are how you set a GPU memory limit and an SLA without hand-waving.
- Getting them wrong looks like a model-quality bug: timeouts, truncated answers, and OOMs.
The parameters
| Parameter | Meaning |
|---|---|
| Max input length | The longest prompt you will accept, in tokens. Longer prompts are rejected or truncated by policy, not by accident. |
| Max output length | The most tokens you will generate for one request. This caps both cost and the KV cache growth during decode. |
| Total sequence length | Input plus output. It has to sit inside the model's context limit and inside your memory budget. |
| Batch size | How many sequences are running together in one iteration. |
| Batched tokens | The sum of tokens those sequences contribute to the iteration. A batch of 4 sequences of length 2,000 is 8,000 batched tokens, not '4'. |
| Concurrency | How many requests are in flight in the service, including ones queued and ones inside the GPU batch. |
| Prefill tokens | Prompt tokens processed to build the KV cache. Often parallel across the prompt. |
| Decode tokens | Tokens produced one step at a time, each step reading the cache. |
Max input length
MeaningThe longest prompt you will accept, in tokens. Longer prompts are rejected or truncated by policy, not by accident.Max output length
MeaningThe most tokens you will generate for one request. This caps both cost and the KV cache growth during decode.Total sequence length
MeaningInput plus output. It has to sit inside the model's context limit and inside your memory budget.Batch size
MeaningHow many sequences are running together in one iteration.Batched tokens
MeaningThe sum of tokens those sequences contribute to the iteration. A batch of 4 sequences of length 2,000 is 8,000 batched tokens, not '4'.Concurrency
MeaningHow many requests are in flight in the service, including ones queued and ones inside the GPU batch.Prefill tokens
MeaningPrompt tokens processed to build the KV cache. Often parallel across the prompt.Decode tokens
MeaningTokens produced one step at a time, each step reading the cache.
Two requests can have the same batch size and wildly different batched tokens. Capacity planning that only counts requests will OOM on the long one and look idle on the short ones.
Why the distinction matters
What moves when the numbers move
- Throughput tracks how many tokens the GPU can chew per iteration, which is why batched tokens predict load better than request count.
- Memory tracks the KV cache: longer sequences and more sequences mean more cached K and V.
- Latency rises when a long prefill shares a batch with a short interactive chat, unless the scheduler is smart. The short chat waits on the long prefill.
- Cost follows GPU-time and hosted-token prices. A max output that is ten times larger than any real answer is a blank check.
- You balance the three on purpose. Maximizing throughput with huge batches can violate a time-to-first-token SLA. Minimizing latency with batch size 1 wastes the card.
The rest of this guide is part of Pro
Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.
Still to come · 3 sections
- Batching sequences of different lengths
- Other knobs
- How to tune
Test yourself
Answer these before moving on — recall is what makes it stick.