Design a web crawler (Googlebot-scale).

Answer preview

Politely fetch billions of pages, extract content + outlinks, deduplicate, and feed downstream indexing / analytics.…