Why this matters
- PolicyDesk's prompt rewrite "felt better" and dropped Recall@4 from 0.9 to 0.6. Nobody noticed until the eval set was rerun.
- Interviewers ask for Precision@K, Recall@K, MRR, and at least two RAGAS metrics, with a definition rather than an expansion of the acronym.
- Optimization has four separate piles of knobs: chunks, retrieval, prompts, and generation. Turning all of them at once teaches you nothing.
- Faithfulness is the metric that catches a fluent lie sitting on top of the right paragraph.
What you are optimizing
A good RAG answer is relevant, supported by the retrieved text, and actually correct against the source of truth. Those are three different properties. A relevant paragraph can be outdated. A supported sentence can support the wrong interpretation. An unsupported sentence can still be true by luck, and you still do not want it, because next time the luck will not hold.
Optimization is how you chase those properties without doubling latency and cost on every request. Evaluation is how you know the chase worked.
Where the knobs are
Four layers
- Ingestion. Cleaner rules, chunk size, overlap, metadata, and whether tables are parsed as tables.
- Retrieval. Embedding model, hybrid versus vector, k, filters, query expansion, re-ranking, MMR.
- Prompt. Refusal rule, citation format, few-shot examples, order of chunks.
- Generation. Model choice, temperature, max tokens. A larger model does not repair a missed retrieval.
Change one layer, rerun the same questions, and write down the metric. If you change the embedding model and the prompt on the same day, you will not know which one moved Recall.
Chunking, retrieval, generation
Chunking. Too small and the fact is severed from its subject. Too large and one vector represents three topics, so it matches loosely and wastes prompt tokens. Overlap of roughly 10 to 20 percent, or a fixed 50 to 100 tokens, keeps boundary facts alive. Add metadata (section, version) so filters can do what chunk size cannot. A useful band to test, not a law: about 200 to 500 tokens when the answer is a specific clause, and up toward 800 to 1000 when the answer is a narrative section. The ingestion lesson's "few hundred to a thousand" is the same band seen from the other side.
Retrieval. Hybrid search for acronyms plus paraphrases. Re-rank the top 20 to 50 with a cross-encoder if the right chunk is in the pool but not first. Expand or rewrite the query only when the right chunk is absent. Lower k if the model is distracted. Raise k if Recall@k says the evidence never arrives. Metadata filters are an accuracy tool, not only a security tool.
Generation. Low temperature for policy. A short max token limit. An explicit "use only the context" line. If answers are unfaithful while the right chunk is in the prompt, change the prompt and the model before you touch the index. If the right chunk is not in the prompt, do not touch the model yet.
The rest of this guide is part of Pro
Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.
Still to come · 4 sections
- Did we fetch the right chunk?
- Did the answer behave?
- A tiny scorecard
- How to tune without thrashing
Test yourself
Answer these before moving on — recall is what makes it stick.