Katkat
Back

Design A Scalable Web Crawler

MicrosoftSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSystem Design

Last updated

This question evaluates scalable system architecture and distributed systems competencies, specifically testing web crawler concepts such as URL discovery and deduplication, scheduling and prioritization, politeness-aware fetching, parsing and link extraction, storage of content and metadata, failure handling, and monitoring.

Design a scalable web crawler that discovers, fetches, parses, and stores web pages for downstream use such as search indexing or content analysis.

Your design should cover:

  • URL discovery and deduplication.
  • Scheduling and prioritization of crawl jobs.
  • Fetching pages while respecting robots.txt, rate limits, and per-domain politeness.
  • Parsing pages and extracting links.
  • Storage for crawled content, metadata, and crawl state.
  • Handling failures, retries, duplicate content, and very large scale.
  • Monitoring and operational concerns.