RAG across Files, the Web, Databases, and APIs

One pipeline, many front doors

Each source becomes documents with page_content and metadata. After that, chunk, embed, and upsert are shared. Do not build a second vector collection per format unless the embedding model or the access domain truly differs.

Loaders, one by one

PDF. PyPDFLoader reads a digital text layer. It will return empty pages for a scan. For scans, OCR first. For multi-column layouts and tables, use a parser that understands layout (the tables lesson). Always keep page number in metadata.

from langchain_community.document_loaders import PyPDFLoader

pages = PyPDFLoader("leave-policy.pdf").load()
print(len(pages), pages[0].metadata.get("page"))

Plain text. TextLoader is enough for markdown and .txt. You still chunk, because a whole wiki export is not one retrieval unit.

CSV and Excel. Each row should become text a human would say, not a comma salad.

from langchain_community.document_loaders import CSVLoader

rows = CSVLoader("holiday-calendar.csv").load()

Prefer a template: "Holiday: Republic Day. Date: 26 January. Offices: all India." Embed that sentence. Keep the date and region as metadata so you can filter "Mumbai office" without hoping the vector notices a column. Spreadsheet loaders (UnstructuredExcelLoader and its cousins) exist for xlsx. Merged cells and multiple sheets still need a look with your eyes.

Web. WebBaseLoader fetches a URL. Strip navigation, cookie banners, and footers before chunking, or every page looks like the site chrome.

from langchain_community.document_loaders import WebBaseLoader

docs = WebBaseLoader("https://intranet.northline.example/holidays").load()

Respect robots, auth, and copyright. An internal crawler with a service account is not a license to index the public internet. Record the URL and the fetch time in metadata so you can expire stale pages.

SQL. Select the columns that belong in the answer. Do not SELECT * from a table of national IDs and embed it.

from langchain_community.utilities import SQLDatabase

db = SQLDatabase.from_uri("postgresql://policy_reader@db/hr")
# Prefer an explicit query that projects safe columns only.
rows = db.run("SELECT name, days, eligibility FROM leave_rules")

Turn each row into a short document, with the primary key in metadata so an update can delete the old point. A database is also a candidate for not using vectors: if the question is "how many days is casual leave?" and the value is a column, a SQL query is the accurate tool. Vectors are for fuzzy wording over text. The text-to-SQL project in the next lesson is that other tool. Use both when the product has both fuzzy policy prose and exact tables.

APIs. Call the endpoint, keep only the fields you will show, and shape them into text plus metadata.

import requests

payload = requests.get(
    "https://hr.internal.example/policies/leave",
    timeout=10,
).json()
text = payload["summary"]
metadata = {"source": "leave-api", "version": payload["version"]}

Paginate. Time out. Do not embed error bodies ({"error": "unauthorized"}) or you will retrieve them forever.