Why this matters
- PolicyDesk's design review failed when the engineer described RAG as "we call the LLM with the PDF." The panel wanted the boxes: load, split, embed, store, search, augment, generate.
- You cannot debug a wrong leave answer until you know which box failed.
- The same six steps work for HR policy, a support bot, or a research assistant. Only the loaders change.
- This is the diagram interviewers draw on the whiteboard before they ask about chunk size.
What the architecture is for
RAG combines information retrieval with generative models. Retrieval finds a small set of passages that are actually about the question. Generation writes the reply. The LLM is allowed to use private or domain text without that text having been baked into the weights.
The system is a pipeline with independent stages. You can swap the PDF loader, the embedding model, or the chat model without rewriting the others. That modularity is the reason production systems do not put "call the model" in one function with the file parser.
What you are building
- A way to pull relevant text from files, sites, databases, or APIs.
- A prompt that places that text next to the user's question.
- A generator that writes the final answer from the augmented prompt.
- An index you can rebuild when the handbook changes, without retraining.
Two phases, not one request
Most of the work happens before anyone asks a question. That is the indexing phase. The query phase is the short path the user waits on.
If you embed the whole handbook on every question, latency and cost collapse the product. Embed documents when they change. Embed only the question at request time.
High-level query path
The six steps
Step by step
- 1. User query. The person asks in normal language: "What is the leave policy at Northline Bank?" No schema, no keywords required, though keywords still help retrieval later.
- 2. Query embedding. A model turns that sentence into a vector, a list of numbers. The same embedding model must be used for chunks and for queries. Mixing
text-embedding-3-smallon the index and another model on the query puts them in different spaces. Search then returns neighbors that are not neighbors. - 3. Document retrieval. The vector database finds chunks whose vectors are close to the query vector. "Close" is a similarity metric, usually cosine. You ask for the top k, often 3 to 8, not the whole corpus.
- 4. Context preparation. You do not paste raw search hits blindly. You keep the text, attach the source, drop duplicates, and respect a token budget. A chunk that is rank 1 but from an archived policy can be filtered out here.
- 5. Prompt augmentation. The user question and the prepared chunks are placed in a template. The template also carries instructions: answer only from the context, cite the source, say you do not know if the context is silent.
- 6. Answer generation. The LLM writes the final answer. It should be a reader of the context, not a second retriever. If the context lacks the answer, the correct generation is a refusal, not a guess.
The full indexing pipeline
Between "we have PDFs" and "we can search" there is a preparation line. Skip a stage and retrieval quality drops in a way that looks like a model bug.
| Stage | What happens | PolicyDesk example |
|---|---|---|
| Load | Read bytes from the source and extract text | Open the HR handbook PDF |
| Split | Break the file into sections before chunking | One document per policy chapter |
| Clean | Strip headers, footers, page numbers, extra whitespace | Remove the repeated 'Northline confidential' footer |
| Chunk | Cut text into retrieval-sized pieces with overlap | About 500 tokens, overlap so a rule is not sliced off its exception |
| Metadata | Attach source, page, title, department, version | department=HR, version=2026-03 |
| Embed | Convert each chunk to a vector | 1536 numbers from the embedding model |
| Store | Upsert vector plus payload into the index | Qdrant collection hr_docs |
Load
What happensRead bytes from the source and extract textPolicyDesk exampleOpen the HR handbook PDFSplit
What happensBreak the file into sections before chunkingPolicyDesk exampleOne document per policy chapterClean
What happensStrip headers, footers, page numbers, extra whitespacePolicyDesk exampleRemove the repeated 'Northline confidential' footerChunk
What happensCut text into retrieval-sized pieces with overlapPolicyDesk exampleAbout 500 tokens, overlap so a rule is not sliced off its exceptionMetadata
What happensAttach source, page, title, department, versionPolicyDesk exampledepartment=HR, version=2026-03Embed
What happensConvert each chunk to a vectorPolicyDesk example1536 numbers from the embedding modelStore
What happensUpsert vector plus payload into the indexPolicyDesk exampleQdrant collection hr_docs
Cleaning matters more than people admit. A footer repeated on every page becomes a chunk that matches almost every query, because it is on every page. You then retrieve the footer instead of the leave table.
Prompt shape
The augmented prompt has three jobs: role, rules, and evidence.
You are PolicyDesk, Northline Bank's HR assistant.
Answer the question using only the context below.
If the answer is not in the context, say you do not know.
Cite the source title and page.
Context:
{retrieved chunks}
Question:
{user question}
Answer:
The braces here are slots your code fills. They are not something the user types. Keep the context visually separated from the question so the model can tell evidence from the ask. That separation also matters for safety when the document itself contains instructions, which is a later topic.
Worked example
Question: "What is our leave policy?"
A good answer names the numbers that are in the chunk and points at the file: "Casual leave is 8 days and privilege leave is 18 days, from Leave Policy, page 5, March 2026 edition." A bad answer adds a carry-forward rule the chunk never mentioned. That extra sentence is the model leaving the context. The template exists to make that departure less likely.
The components
| Component | Responsibility |
|---|---|
| Data loader | Read PDF, HTML, office files, databases, or APIs into text plus metadata |
| Text splitter | Cut documents into chunks small enough to embed and retrieve |
| Embedding model | Turn text into vectors with a fixed dimension |
| Vector database | Store vectors and search the nearest ones, with filters on metadata |
| Prompt augmentation | Build the final prompt from the question, the chunks, and the rules |
| LLM | Generate the answer from that prompt |
Data loader
ResponsibilityRead PDF, HTML, office files, databases, or APIs into text plus metadataText splitter
ResponsibilityCut documents into chunks small enough to embed and retrieveEmbedding model
ResponsibilityTurn text into vectors with a fixed dimensionVector database
ResponsibilityStore vectors and search the nearest ones, with filters on metadataPrompt augmentation
ResponsibilityBuild the final prompt from the question, the chunks, and the rulesLLM
ResponsibilityGenerate the answer from that prompt
Add two more boxes in any system you expect to operate: an evaluation set (questions with known good sources) and a citation renderer (show the user which chunk was used). They are not required to draw the happy path. They are required to know if the happy path is lying.
Why this shape wins
Benefits
- Answers can track documents that change monthly, without a training run.
- Private text stays in your index. You send only the retrieved snippets to the model API, which is still a privacy decision you must make deliberately.
- Domain accuracy improves because the domain words are in the prompt, not hoped for in the weights.
- You can explain an answer by showing the chunk. Trust follows the citation.
- You can scale the corpus by scaling the index, and scale traffic by scaling the stateless query service.
Where people use it
Common use cases
- Company knowledge and HR policy. Handbooks, benefits, holiday calendars. PolicyDesk is this case.
- Customer support. Past resolutions and product docs, so the bot stops inventing refund rules.
- Research assistants. Papers and internal notes, with citations the reader can open.
- Enterprise search. Find the paragraph, then summarize it, instead of returning ten blue links alone.
- Legal, healthcare, and finance. High-stakes text where an unsourced sentence is a defect, not a style choice. These domains also need access control and a human for decisions the document does not settle.
Quick recall
Everything you need if you only revisit this box.
- RAG is retrieval plus generation, connected by a prompt.
- Index documents ahead of time. At query time, embed only the question.
- Six query steps: question, query embedding, retrieval, context prep, prompt augmentation, generation.
- Indexing stages: load, split, clean, chunk, metadata, embed, store.
- Same embedding model for documents and queries.
- Use cases include policy, support, research, enterprise search, and regulated domains that need citations.
- Debug by asking which stage produced the wrong text.
Test yourself
Answer these before moving on — recall is what makes it stick.