Why this matters
- PolicyDesk's notebook answered correctly and could not survive two simultaneous users, a restart, or a handbook update.
- Interviewers ask where the index lives, what is cached, and how you know the answer was faithful last Tuesday.
- The model API key, the vector database, and the documents are three different secrets and three different failure domains.
- Latency has two budgets: retrieval and generation. You cannot fix generation latency by adding chunk overlap.
What "production" has to guarantee
Requirements before you call it launched
- Availability. The API stays up when one worker dies. The index has a disk and a backup.
- A bounded latency. Users will wait for a short answer. They will not wait for you to re-embed the PDF inside the request.
- Safe ingestion. New documents enter through a job, are scanned and chunked, and do not block queries.
- Caching. Repeated questions and hot chunks should not call the chat model every time, within a freshness rule.
- Observability. You can see latency, error rate, retrieval ids, token usage, and a sample of refusals.
- A boring release. You can roll a new prompt or a new index without an outage.
Indexing and serving are separate processes. If they share one function, a 200-page upload stalls everyone else's questions.
A shape you can draw
Redis caches repeated answers. Object storage holds the original PDFs. Workers ingest off to the side.
The request path: authenticate, check a cache key, embed the question, search with the caller's ACL filter, build the prompt, call the model, store a cache entry if the policy allows, return the answer and the citations. No loader. No splitter.
A small API
from fastapi import FastAPI
from pydantic import BaseModel, Field
app = FastAPI()
class Ask(BaseModel):
question: str = Field(min_length=1, max_length=500)
@app.post("/ask")
def ask(body: Ask):
cached = cache.get(body.question)
if cached:
return cached
docs = retriever.invoke(body.question)
answer = generate(body.question, docs)
payload = {
"answer": answer,
"sources": [d.metadata.get("source") for d in docs],
}
cache.set(body.question, payload, ttl_seconds=300)
return payload
Validate length. A 100,000-character "question" is a bill and a prompt-injection attempt. Return sources every time so the UI can show them. Do not cache personalized answers (answers that depend on the caller's identity or live probation status) under a key that is only the question text. You would serve one employee's answer to another.
Run several workers behind the load balancer. The process must be stateless: no handbook stored in a global variable that only one worker loaded.
The rest of this guide is part of Pro
Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.
Still to come · 5 sections
- Container
- Kubernetes, in one picture
- Caching
- Observability
- Practices
Test yourself
Answer these before moving on — recall is what makes it stick.