Skip to main content
PreparationZone Logo
PrepZone

Web Crawler Design

Discover, fetch and index billions of pages with politeness, deduplication and distributed workers.

Pass checkpoint below

Read these first

Design StreamIndex — StreamHub's web crawler that discovers streamer profile pages across the internet, extracts metadata, and feeds the search index. Billions of pages, distributed workers, and strict politeness constraints.

Requirements

Functional requirements

  • Given a seed URL, discover and fetch linked pages up to N hops deep.
  • Extract structured data: title, description, Open Graph tags, links.
  • Store raw HTML and parsed metadata for the search indexer.
  • Respect robots.txt and crawl-delay directives.
  • Deduplicate URLs — never fetch the same canonical URL twice.

Non-functional requirements

  • Scale: Crawl 1B pages/month; 10B total indexed pages.
  • Politeness: Max 1 request/sec per domain (configurable).
  • Freshness: Re-crawl popular pages every 24 h; long-tail every 30 days.
  • Availability: 99% — crawler downtime delays index freshness, not user-facing latency.
  • Storage: Raw HTML averages 50 KB/page → 10B × 50 KB = 500 TB compressed.

Out of scope: JavaScript rendering (headless browser farm is v2), login-gated pages, real-time crawl.

Estimation

  • 1B pages/month ÷ 30 days ÷ 86400 ≈ 400 pages/sec sustained crawl rate.
  • At 1 req/sec/domain limit: need 400+ domains crawled in parallel simultaneously.
  • URL frontier: 10B URLs × 100 bytes ≈ 1 TB for URL store (Bloom filter + disk).
  • Bandwidth: 400 pages/sec × 50 KB ≈ 20 MB/sec ≈ 1.7 TB/day ingress.

Web crawler architecture

COMPUTE
EventBridgescheduler
INTEGRATION
SQS frontierURL queue
COMPUTE
Crawler workersLambda / EKS
STORAGE
Amazon S3raw pages
ANALYTICS
OpenSearchindex
EventBridge scheduler → SQS → workers → S3 raw HTML → OpenSearch index.

The rest of this guide is part of Pro

Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.

Still to come · 4 sections

  • API design
  • High-level architecture
  • Deep dive: URL frontier and politeness
  • Deep dive: deduplication and content change detection

Test yourself

Answer these before moving on — recall is what makes it stick.