Design a Fault-Tolerant Wikipedia Archiver

Question

Design a distributed system that archives all Wikipedia pages and keeps the archive current. You have one reliable primary machine, about one thousand unreliable and untrusted worker bots with different IP addresses, and persistent blob storage. Focus on discovery, deduplication, task assignment, worker failure, safe storage, and periodic recrawling rather than a reader-facing website.

Constraints & Assumptions

• Workers may disappear, retry, lie about completion, or return malformed content.
• The primary coordinates metadata but cannot proxy every page body.
• Pages link to other pages, and the frontier is much larger than memory.
• Fetches must respect site policy, pacing, and robots requirements.
• An archived version is published only after content and provenance checks pass.

Clarifying Questions to Ask

• Are historical versions required, or only the latest snapshot? Preserve versioned snapshots when content changes.
• May official dumps or APIs be used? Prefer them where allowed; crawling fills freshness or coverage gaps.
• What freshness target and total bandwidth budget apply?
• How is worker trust established, and which results require independent verification?
• Are media assets in scope? Treat them as a separate typed crawl with its own policy.