Why this matters
- PolicyDesk retrieved the page footer for "how many casual leaves?" because the footer was on every page and the leave table was split across two chunks with no overlap.
- Interviewers ask "how do you chunk?" to see if you have more than one strategy and a reason to pick it.
- Metadata is how you later filter "HR only" or "current version only." Without it, every department's PDF competes in one pile.
- Loaders differ by format. A PDF loader that works on a digital handbook fails on a scanned one.
What ingestion is
Ingestion is the path from a source file or record into the vector index. You load text from PDFs, office files, sites, APIs, and databases. You clean it, split it into chunks, attach metadata, embed the chunks, and store them.
The quality of every later answer is bounded by this path. A brilliant prompt cannot quote a paragraph you never stored. A fresh embedding cannot repair a chunk that starts mid-sentence and ends mid-table.
Pipeline
Seven processing steps
- Load. Pick a loader that understands the format. A PDF loader, a web loader, and a SQL loader do not produce the same kind of text.
- Extract. Pull the words out. Digital PDFs yield a text layer. Scans need OCR. Slides and sheets need a loader that keeps headings and cells, not a dump of every string on the page.
- Clean. Remove running headers, footers, page numbers, leftover hyphenation, and repeated navigation. Collapse whitespace. Drop chunks that are only a copyright line.
- Chunk. Split the cleaned text into pieces of a chosen size, with a chosen overlap.
- Add metadata. Source, page, title, section, chunk id, department, version, updated time. This is the payload you filter on later.
- Embed. Send each chunk through the embedding model. One vector per chunk.
- Store. Write the vector and the payload together. The text itself must be in the payload or in a store you can join back by id. A vector without the original text cannot be shown to the LLM.
Sources you will actually meet
| Source | Examples | Watch-out |
|---|---|---|
| Documents | PDF, DOCX, TXT, PPTX, XLSX | Scanned PDFs have no text layer |
| Web | URLs, help centers, blogs, internal docs sites | Boilerplate nav and cookie banners |
| Databases | MySQL, Postgres, MongoDB | Rows are structured; decide which columns form the text |
| APIs | REST responses, internal services | Paginate, and respect auth and rate limits |
| Cloud files | S3, Drive, SharePoint | Permissions must be enforced again at query time |
| Other | Email, Notion, Confluence | HTML leftovers and user-written instructions inside the page |
Documents
ExamplesPDF, DOCX, TXT, PPTX, XLSXWatch-outScanned PDFs have no text layerWeb
ExamplesURLs, help centers, blogs, internal docs sitesWatch-outBoilerplate nav and cookie bannersDatabases
ExamplesMySQL, Postgres, MongoDBWatch-outRows are structured; decide which columns form the textAPIs
ExamplesREST responses, internal servicesWatch-outPaginate, and respect auth and rate limitsCloud files
ExamplesS3, Drive, SharePointWatch-outPermissions must be enforced again at query timeOther
ExamplesEmail, Notion, ConfluenceWatch-outHTML leftovers and user-written instructions inside the page
A loader's job is text plus location. PyPDFLoader is a reasonable default for digital PDFs. WebBaseLoader fetches a URL. CSVLoader and spreadsheet loaders keep rows. SQLDatabaseLoader turns selected rows into documents. TextLoader is enough for plain files. None of these are magic on a scanned invoice or a merged-cell report. Those need a table strategy, covered in the tables lesson.
Four ways to chunk
| Strategy | How it splits | Use it when |
|---|---|---|
| Fixed size | Every N tokens, often with overlap | You want a predictable prompt budget and the text has no reliable structure |
| Recursive | Try paragraphs, then lines, then sentences, then tokens, until pieces fit | Prose and handbooks. This is the default for PolicyDesk |
| Semantic | Split where the meaning shifts, using embedding similarity between neighboring sentences | Long essays where a paragraph boundary is a topic boundary |
| Sliding window | Overlapping windows so a fact on the boundary appears in two chunks | You have measured that rules are being cut in half |
Fixed size
How it splitsEvery N tokens, often with overlapUse it whenYou want a predictable prompt budget and the text has no reliable structureRecursive
How it splitsTry paragraphs, then lines, then sentences, then tokens, until pieces fitUse it whenProse and handbooks. This is the default for PolicyDeskSemantic
How it splitsSplit where the meaning shifts, using embedding similarity between neighboring sentencesUse it whenLong essays where a paragraph boundary is a topic boundarySliding window
How it splitsOverlapping windows so a fact on the boundary appears in two chunksUse it whenYou have measured that rules are being cut in half
Recursive splitting is the workhorse because it prefers natural boundaries. It tries to break on \n\n, then \n, then spaces, and only then cuts a token window. A leave rule that is one paragraph stays one paragraph if it fits.
Fixed-size splitting is honest and blunt. It will cut a sentence. Overlap exists so the next window still contains the end of that sentence.
Semantic splitting costs an embedding call during ingestion. It pays off when a single heading hides three topics, and it wastes money when every paragraph is already one topic.
Sliding-window overlap is not a fifth religion. It is a parameter on the other strategies. The notes you revise from often start at chunk_size=500 and chunk_overlap=50. Read those numbers carefully: many splitters count characters unless you pass a token-length function. Five hundred characters is a short paragraph. Five hundred tokens is most of a page. Pick the unit on purpose.
A handbook cut into cards
Sample paragraph:
Northline Bank provides savings accounts, current accounts, personal loans, home loans, and credit cards to customers.
With a tiny chunk size for the demo, that becomes:
| Chunk | Text |
|---|---|
| 1 | Northline Bank provides savings accounts, current accounts, home loans |
| 2 | personal loans, home loans and credit cards |
| 3 | services to its customers |
1
TextNorthline Bank provides savings accounts, current accounts, home loans2
Textpersonal loans, home loans and credit cards3
Textservices to its customers
Chunk 1 lost "credit cards." Chunk 2 repeats "home loans" because of overlap, which is what you want on a boundary, and it dropped "personal loans" from the start if the window moved too far. Chunk 3 is a tail with no subject. This is why tiny chunks hurt. The embedding for chunk 3 is an embedding of "services to its customers," which matches almost any product question and answers none of them.
For the real handbook, keep the leave section intact: the day counts, the eligibility sentence, and the exception for probation, in one chunk or in overlapping chunks that each still contain the day counts.
Metadata
{
"source": "leave-policy.pdf",
"page": 5,
"chunk_id": "chunk_1",
"title": "Leave Policy",
"department": "HR",
"section": "Casual leave",
"version": "2026-03",
"file_type": "pdf"
}
Why each field exists
- source and page let you cite and let a human open the file.
- chunk_id lets you delete or replace one piece when a paragraph changes.
- title and section improve the prompt. The model reads "Casual leave" even if the chunk body starts mid-list.
- department is the filter for "only HR" so a retail-product PDF does not answer a leave question.
- version is how you prefer the March handbook over the archived one.
- file_type helps you debug a loader bug ("all the bad hits are pptx").
Store the chunk text in the payload too. Searching returns ids and vectors; the generator needs words.
What the code is doing
The shape, independent of which LangChain release you pin:
from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
loader = PyPDFLoader("leave-policy.pdf")
documents = loader.load()
splitter = RecursiveCharacterTextSplitter(
chunk_size=500,
chunk_overlap=50,
)
chunks = splitter.split_documents(documents)
for chunk in chunks:
chunk.metadata["department"] = "HR"
chunk.metadata["version"] = "2026-03"
load() yields one document per page for many PDF loaders, with page already in metadata. The splitter keeps that metadata on child chunks. If you split and then forget to copy metadata, every chunk looks the same and filters stop working.
Practices that prevent silent failure
Ingestion habits
- Use a loader that matches the format. Do not run a PDF loader on HTML and hope.
- Clean headers, footers, and nav before you embed. They dominate similarity if you leave them in.
- Choose chunk size for the questions you expect, then read twenty random chunks with your eyes.
- Overlap on purpose so a rule and its exception can co-occur.
- Treat tables and multi-page grids as their own path. Character splitting destroys rows.
- Write metadata at ingest. You cannot filter on a field you never stored.
- Re-embed when the source changes. An old vector is a stale answer with a fresh timestamp on the API call.
Common failure modes
- Complex layouts. Multi-column PDFs interleave two columns into one nonsense paragraph.
- Irregular rows. Tables with merged cells become a soup of numbers.
- Chunk too small. The fact is cut away from the noun it describes.
- Chunk too large. Several topics share one vector, so the hit is vaguely related and the prompt is expensive.
- No overlap. The answer sat on the boundary and is in neither chunk completely.
- Huge corpora with no filter. Retrieval time and false neighbors both grow. Metadata and a sensible top-k are the first controls, not a bigger model.
Quick recall
Everything you need if you only revisit this box.
- Ingestion is load, extract, clean, chunk, metadata, embed, store.
- Sources include documents, web pages, SQL, APIs, cloud drives, and wikis. Each needs a fitting loader.
- Fixed, recursive, semantic, and sliding-window overlap are the four strategies. Recursive is the default for prose.
- Start near a few hundred to a thousand tokens with overlap, then measure. Know whether your splitter counts characters or tokens.
- Metadata (source, page, department, version, chunk id) is what you filter and cite with.
- Clean footers or they become the top hit for every query.
- The chunk text must be stored, not only the vector.
Test yourself
Answer these before moving on — recall is what makes it stick.