Why this matters
- Northline Bank asked PolicyDesk "what is our leave policy?" and the model invented a 15-day casual-leave rule that does not exist in the handbook.
- Interviewers use this failure to test whether you know the six hard limits of an LLM used alone.
- Every later lesson — chunking, embeddings, vector search, prompts — exists to close one of those limits.
- RAG is the default fix for private, changing, citable knowledge. Fine-tuning is a different tool.
What an LLM actually has
An LLM is a next-token predictor trained on a snapshot of text. At inference it has three things: the weights, the current prompt, and whatever it generates next. It does not have a live connection to your HR drive, yesterday's product change, or a citation trail, unless you build one.
PolicyDesk's first prototype sent the user question straight to the model. That is LLM alone. It is fast and simple, and it is wrong the moment the answer lives outside the weights.
Six limitations of an LLM used alone
- Knowledge cutoff. Training stops on a date. The model does not know policy edits, library releases, or prices that landed after that snapshot. "Latest" in the weights means "latest in the training mix," which is already stale.
- Hallucinations. The model can produce fluent, specific, false text: a leave rule, a paper title, an API parameter. It sounds reasonable because fluency is what training rewarded. Truth was never a separate store.
- No access to private or live data. Internal PDFs, databases, tickets, and today's production metrics are invisible. The model cannot open
leave-policy.pdfunless your system puts that text in the prompt. - Weak source traceability. Even a correct-sounding answer does not say which paragraph it came from. An auditor cannot tell a remembered fact from a guess.
- Context-window limits. The model can read only so many tokens at once. A 400-page handbook, a year of tickets, or a full codebase will not fit. Pasting "everything" also raises cost and hides the relevant page in the middle of a huge prompt.
- Uneven domain accuracy. General web text trains a general voice. Niche policy, a bank's product names, or a medical protocol the model barely saw will be answered poorly unless you add that material.
LLM alone versus RAG
| Aspect | LLM alone | RAG |
|---|---|---|
| Knowledge | Only what was compressed into the weights | Weights plus documents fetched for this question |
| Freshness | Frozen at the training cutoff | As fresh as the index you maintain |
| Private data | Unavailable | Your PDFs, wiki, database, and tickets, if you index them |
| Citations | No source pointer | You can return the chunk, page, and file |
| Hallucinations | The model fills gaps from patterns | Fewer gaps, if the right chunk was retrieved and the prompt forbids guessing |
| Failure mode | Confident invention | Wrong or empty retrieval, or the model ignoring the context |
Knowledge
LLM aloneOnly what was compressed into the weightsRAGWeights plus documents fetched for this questionFreshness
LLM aloneFrozen at the training cutoffRAGAs fresh as the index you maintainPrivate data
LLM aloneUnavailableRAGYour PDFs, wiki, database, and tickets, if you index themCitations
LLM aloneNo source pointerRAGYou can return the chunk, page, and fileHallucinations
LLM aloneThe model fills gaps from patternsRAGFewer gaps, if the right chunk was retrieved and the prompt forbids guessingFailure mode
LLM aloneConfident inventionRAGWrong or empty retrieval, or the model ignoring the context
RAG does not retrain the model. It changes what the model is allowed to read for this question.
What RAG adds
- It retrieves relevant information from an external index instead of hoping the weights remember it.
- It combines that text with the user question in one prompt.
- The generated answer can be more accurate and more up to date than the weights alone.
- The same response can carry a source: file name, page, section, URL.
How each limit is closed
| Limitation | How RAG helps |
|---|---|
| Outdated knowledge | Search the latest handbook, policy, or site, then put those passages in the prompt. |
| No private data | Index company documents, a knowledge base, or a database extract the model was never trained on. |
| Hallucinations | Constrain the answer to the retrieved text and tell the model to say it does not know when the context is silent. |
| No source | Return the chunk ids, file names, and pages that were actually used. |
| Weak domain accuracy | Retrieve the domain document — leave rules, dosage tables, API specs — instead of asking the model to recall a niche. |
Outdated knowledge
How RAG helpsSearch the latest handbook, policy, or site, then put those passages in the prompt.No private data
How RAG helpsIndex company documents, a knowledge base, or a database extract the model was never trained on.Hallucinations
How RAG helpsConstrain the answer to the retrieved text and tell the model to say it does not know when the context is silent.No source
How RAG helpsReturn the chunk ids, file names, and pages that were actually used.Weak domain accuracy
How RAG helpsRetrieve the domain document — leave rules, dosage tables, API specs — instead of asking the model to recall a niche.
RAG is a retrieval fix. It does not enlarge the model's raw reasoning, and it does not replace evaluation.
The context window is only partly solved. RAG selects a handful of chunks so the prompt stays small. It does not let you dump a warehouse into one call. If retrieval misses the right page, the window is full of the wrong pages.
The leave-policy example
A Northline employee asks: "What is our leave policy?"
Alone, the model might say casual leave is 12 days because that number is common on the public web. Northline's handbook says 8 days of casual leave and 18 days of privilege leave, updated in March. RAG embeds the question, finds the March handbook chunk, and the prompt says: answer only from this text, and quote the section.
That is the whole product difference. The model did not learn Northline's policy. The index did.
RAG still fails
Retrieval does not make the model honest by magic. These four failures show up in every real assistant:
Limits that remain
- Stale documents. If the indexed PDF is last year's handbook, the answer is confidently stale. RAG is only as fresh as the pipeline that re-embeds changes.
- Missed retrieval. If the right paragraph is never in the top-k, the model is answering from the wrong pages or from nothing. Similarity search can miss a page that uses different words ("PTO" versus "privilege leave").
- Ignored context. The model sometimes answers from its priors even when the chunk is sitting in the prompt. A strict instruction ("if it is not in the context, say you do not know") and a low temperature reduce this. They do not erase it.
- Hallucinations shrink; they do not disappear. Numbers, names, and eligibility rules still get smoothed. Citations can be attached to the wrong sentence. You still need checks.
What to remember
Key takeaways
- LLMs are strong writers and weak databases. They do not look up your files unless you build the lookup.
- RAG retrieves relevant, real text at query time and asks the model to use it.
- Quality of the corpus beats size of the model for company-specific questions. A clean, current handbook beats a larger model guessing.
- Treat "I don't know" as a successful answer when the index has no supporting chunk.
Quick recall
Everything you need if you only revisit this box.
- Six LLM limits: cutoff, hallucination, no private data, no traceable source, finite context, weak niche accuracy.
- LLM alone reads the weights. RAG reads the weights plus retrieved chunks.
- The leave-policy question is the canonical demo: guessing a number versus quoting the handbook.
- RAG fails when documents are stale, retrieval misses, or the model ignores the prompt.
- RAG reduces hallucinations. It does not eliminate them.
- Indexing documents is not the same thing as fine-tuning the model.
Test yourself
Answer these before moving on — recall is what makes it stick.