Design StreamIndex — StreamHub's web crawler that discovers streamer profile pages across the internet, extracts metadata, and feeds the search index. Billions of pages, distributed workers, and strict politeness constraints.
Requirements
Functional requirements
- Given a seed URL, discover and fetch linked pages up to N hops deep.
- Extract structured data: title, description, Open Graph tags, links.
- Store raw HTML and parsed metadata for the search indexer.
- Respect robots.txt and crawl-delay directives.
- Deduplicate URLs — never fetch the same canonical URL twice.
Non-functional requirements
- Scale: Crawl 1B pages/month; 10B total indexed pages.
- Politeness: Max 1 request/sec per domain (configurable).
- Freshness: Re-crawl popular pages every 24 h; long-tail every 30 days.
- Availability: 99% — crawler downtime delays index freshness, not user-facing latency.
- Storage: Raw HTML averages 50 KB/page → 10B × 50 KB = 500 TB compressed.
Out of scope: JavaScript rendering (headless browser farm is v2), login-gated pages, real-time crawl.
Estimation
- 1B pages/month ÷ 30 days ÷ 86400 ≈ 400 pages/sec sustained crawl rate.
- At 1 req/sec/domain limit: need 400+ domains crawled in parallel simultaneously.
- URL frontier: 10B URLs × 100 bytes ≈ 1 TB for URL store (Bloom filter + disk).
- Bandwidth: 400 pages/sec × 50 KB ≈ 20 MB/sec ≈ 1.7 TB/day ingress.
Web crawler architecture
The rest of this guide is part of Pro
Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.
Still to come · 4 sections
- API design
- High-level architecture
- Deep dive: URL frontier and politeness
- Deep dive: deduplication and content change detection
Test yourself
Answer these before moving on — recall is what makes it stick.