Document Summarization Pipeline

Why this project

Summarization teaches token limits, chunking, and map-reduce — foundational LLM engineering skills.

Tech stack

Skills demonstrated

• Map-reduce summarization
• Batch processing
• Token management
• Versioning

Interview talking points

• Map-reduce walkthrough
• Context window limits
• Summary drift on re-run
• Cost per document

Problem statement

Batch and on-demand summarization of long documents (reports, contracts). Map-reduce for docs exceeding context window. Store summaries with link to source version.