Why this matters
- PolicyDesk occasionally answered "casual leave casual leave casual leave" or restated the same sentence until max tokens.
- It looks like a bug in the handbook. It is usually a decoding and prompt problem.
- Interviewers ask for temperature, top-p, and a repetition penalty, plus the damage a harsh penalty does to legitimate repeated terms.
- The fix is a small set of knobs and a better stop condition, not a new vector database.
Why the loop starts
The model samples a next token from a probability distribution. If a small set of tokens is much more likely than the rest, sampling keeps picking them. After it has emitted "casual leave," those tokens are in the context, and for some prompts the distribution puts them at the top again. Autoregression then reinforces itself.
Conditions that make it worse
- Training patterns. Some public text is repetitive (lists, chants, boilerplate). The model learned that repetition is a possible continuation.
- Unclear prompts. "Explain the policy" with a huge context and no length cap invites the model to keep talking after it has finished.
- Greedy or low-diversity decoding. Greedy search and very small top-k always take the peak. If the peak is the phrase just emitted, you get a loop. High temperature can also wander into a repeated rhythm. Both extremes show up.
- No penalty. If the decoder does not down-weight tokens it recently used, nothing pushes it off the loop.
- Long max tokens. The loop needs room. A 1,000-token cap on an answer that should be 80 words gives it that room.
Repetition can be a stuck token ("the the the") or a stuck sentence (the same leave rule three times). Sentence-level loops are the ones users notice in a RAG answer. Token-level loops show up more often when the model is uncertain or the prompt is degenerate.
The rest of this guide is part of Pro
Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.
Still to come · 3 sections
- Knobs that break the loop
- A picture of the controls
- What you trade away
Test yourself
Answer these before moving on — recall is what makes it stick.