Katkat
Back

Design a large-scale web crawler

MetaSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

This question evaluates a candidate's ability to design scalable, fault-tolerant distributed systems for web-scale data acquisition, covering competencies such as crawler architecture, URL frontier management, deduplication, per-domain rate limiting and robots.

Design a web crawler that continuously discovers and downloads web pages.

Requirements:

  • Start from seed URLs and crawl at scale.
  • Avoid duplicate crawling; obey robots.txt and per-domain rate limits.
  • Prioritize important/fresh pages.
  • Store fetched content and metadata.
  • Be fault tolerant and resumable.

Out of scope: full search ranking, rendering heavy JS.