Why this matters
- Tool questions in interviews are really boundary questions: what does this component own, and what should it refuse to own?
- PolicyDesk wasted a week because three libraries all wanted to "be the application." One orchestrator is enough.
- You should be able to swap the vector store or the chat model without rewriting the PDF parser.
- Names move fast. The categories below are the stable part.
The map
Orchestration wraps the arrows. Monitoring watches all of them. Agents sit on top of the LLM and call back into search.
Load and split
| Job | Tools people name | Notes |
|---|---|---|
| PyPDF, pdfplumber, Unstructured, Docling | Digital text versus scans versus tables. One library will not win all three. | |
| Web | WebBaseLoader, a real crawler if you must | Clean HTML. Allow-list hosts. |
| Office and text | Unstructured, python-docx, TextLoader | Keep headings. They are retrieval signals. |
| Tables | Camelot, Tabula, pdfplumber | Covered in the tables lesson. Do not character-split a grid. |
| SQL | SQLAlchemy or the LangChain SQL utility | Project safe columns only. |
| Split | RecursiveCharacterTextSplitter, and token-aware variants | Default for prose. Semantic splitters cost an embedding at ingest. |
PDF
Tools people namePyPDF, pdfplumber, Unstructured, DoclingNotesDigital text versus scans versus tables. One library will not win all three.Web
Tools people nameWebBaseLoader, a real crawler if you mustNotesClean HTML. Allow-list hosts.Office and text
Tools people nameUnstructured, python-docx, TextLoaderNotesKeep headings. They are retrieval signals.Tables
Tools people nameCamelot, Tabula, pdfplumberNotesCovered in the tables lesson. Do not character-split a grid.SQL
Tools people nameSQLAlchemy or the LangChain SQL utilityNotesProject safe columns only.Split
Tools people nameRecursiveCharacterTextSplitter, and token-aware variantsNotesDefault for prose. Semantic splitters cost an embedding at ingest.
Embeddings
| Option | When |
|---|---|
| OpenAI text-embedding-3-small or large | Hosted, strong default if the text is allowed to leave your network |
| BGE, E5, or other sentence-transformers | Self-hosted, including on a GPU you already run |
| Cohere Embed | Hosted, including multilingual needs |
| A model already in your cloud (Bedrock, Vertex, Azure OpenAI) | When procurement already standardized there |
OpenAI text-embedding-3-small or large
WhenHosted, strong default if the text is allowed to leave your networkBGE, E5, or other sentence-transformers
WhenSelf-hosted, including on a GPU you already runCohere Embed
WhenHosted, including multilingual needsA model already in your cloud (Bedrock, Vertex, Azure OpenAI)
WhenWhen procurement already standardized there
The orchestrator should not hide which model you called. Log the model id. Your vectors are married to it.
Vector stores
FAISS (library), Chroma (prototype), Qdrant and Weaviate (self-hosted servers, Weaviate especially if you want hybrid in one query), Milvus (large distributed), Pinecone (managed), pgvector (Postgres you already operate). The vector-database lesson has the selection criteria. The ecosystem point: the store does not load PDFs and does not write answers.
Chat models
Hosted: OpenAI GPT models, Anthropic Claude, Google Gemini, Cohere Command. Self-hosted or VPC: Llama family models and others through vLLM, TGI, or Ollama. "Best model" is not a stable sentence. Pick with the eval set: faithfulness and refusal on your questions, then cost and latency. A small model that obeys the context can beat a large model that improvises.
Orchestration
LangChain is a wide toolbox: loaders, prompts, retrievers, agents. It is a reasonable default and it changes quickly. Learn the pipeline more than the import path.
LlamaIndex focuses on indexing and retrieval structures, including the hierarchical ideas in the architecture lesson. Teams pick it when the index, not the agent, is the product.
Haystack is a pipeline framework with explicit nodes. It reads clearly in reviews.
A few dozen lines of Python are a valid orchestrator for PolicyDesk's naive path. Frameworks pay off when you need agents, tracing, and many loaders. They cost you an upgrade treadmill. Do not wrap a framework in another framework.
Agents and tools
LangChain agents, LlamaIndex agents, and smaller libraries (the pattern matters more than the brand) implement the tool loop. Tool protocols such as MCP are one way to expose a search or a database to a model with a described schema. The rule from the agent lesson still holds: specific tools, validated arguments, a step budget.
Semantic kernels and similar enterprise SDKs show up in Microsoft-centric stacks. They are another orchestrator, not a new kind of retrieval.
Deployment and inference servers
FastAPI (or any HTTP framework) is the application boundary. Docker and Kubernetes run it. vLLM, TensorRT-LLM, and TGI (text generation inference) serve open-weight models with batching and KV-cache management. Ollama is a simple local runner, good for a laptop, not your availability story. If you call a hosted chat API, you do not run these servers. You still need timeouts and a budget.
Monitoring and evaluation
| Need | Tools people name |
|---|---|
| Traces of a RAG request | LangSmith, Phoenix / Arize, OpenTelemetry, Langfuse |
| Offline metrics | RAGAS, your own Precision@K harness, DeepEval |
| Service metrics | Prometheus and Grafana, the same stack as the rest of your APIs |
| Logs | Structured JSON to whatever you already use. Include retrieved ids, not only the answer. |
Traces of a RAG request
Tools people nameLangSmith, Phoenix / Arize, OpenTelemetry, LangfuseOffline metrics
Tools people nameRAGAS, your own Precision@K harness, DeepEvalService metrics
Tools people namePrometheus and Grafana, the same stack as the rest of your APIsLogs
Tools people nameStructured JSON to whatever you already use. Include retrieved ids, not only the answer.
A trace that shows the question, the retrieved titles, the prompt, and the answer will debug more incidents than a new vector database. Adopt tracing before you adopt a second orchestration framework.
How to choose without collecting logos
A sane PolicyDesk stack
- Python, FastAPI, and a recursive splitter.
- One embedding model, logged by name.
- Qdrant or pgvector, with payload filters.
- One chat model, temperature 0, behind a timeout.
- LangChain or plain functions, not both wrapped around each other.
- A CSV of labeled questions and a script that prints Recall@4 and a faithfulness sample.
- OpenTelemetry or one of the RAG tracers, plus ordinary API metrics.
Swap a box when the eval set or the ops burden says so. "We use seven AI tools" is not a design.
Quick recall
Everything you need if you only revisit this box.
- Stages: load, split, embed, store, retrieve, generate, trace.
- Loaders differ by format. Splitter default is recursive and token-aware.
- Embeddings are hosted or self-hosted. Vectors are married to the model id.
- Vector stores range from FAISS and Chroma to Qdrant, Weaviate, Milvus, Pinecone, and pgvector.
- LangChain, LlamaIndex, and Haystack orchestrate. Plain Python is allowed.
- vLLM, TGI, and TensorRT-LLM serve open models. Hosted APIs replace them.
- Trace retrieved ids. Evaluate with a fixed question set. Then, and only then, add tools.
Test yourself
Answer these before moving on — recall is what makes it stick.