Katkat
Back

Design a concurrent web crawler

SnowflakeSoftware Engineer · Onsite
HardSoftware EngineerOnsiteSystem Design

Last updated

This problem evaluates your ability to design a distributed, concurrent web crawling architecture that balances high throughput with politeness policies, concurrency control, and robust state management.

Design a Concurrent Web Crawler

Design a distributed web crawler system capable of fetching and indexing millions of pages from the public internet. The system must operate concurrently while adhering to specific constraints.

Requirements

  1. Politeness: You must respect robots.txt files and implement per-host rate limiting (e.g., waiting at least X milliseconds between requests to the same domain).
  2. Deduplication: Ensure the crawler does not process the same URL or content multiple times.
  3. Scalability: The system should be able to scale horizontally. Assume 100 million pages per day.
  4. Robustness: Handle network timeouts, malformed HTML, and site-wide outages gracefully.
  5. URL Normalization: Handle redirects and canonical URL forms (e.g., removing fragments, handling trailing slashes).

Deliverables

  • Provide a high-level system architecture diagram (described in text).
  • Describe the data model for the URL frontier (the queue of URLs to visit).
  • Explain how you manage concurrency across multiple worker nodes while enforcing global and per-domain rate limits.
  • Discuss the storage strategy for crawled content and metadata.

Assumptions

  • You have access to a distributed key-value store (like Redis or Cassandra) and a message broker (like Kafka or RabbitMQ).
  • The focus is on the coordination logic and concurrency strategy, not on the HTML parsing algorithms.

Hints

Hint 1

Use a distributed priority queue for the URL frontier and classify URLs by domain to simplify per-host rate limiting.

Hint 2

Implement a Bloom filter for quick, memory-efficient URL deduplication before hitting your persistent storage.

Hint 3

Consider a producer-consumer model where 'Fetcher' nodes pull from domain-specific queues and a 'Scheduler' process coordinates the distribution of work.

Follow-up questions

  • How would you handle dynamic content that requires JavaScript execution (e.g., React/Vue sites)?
  • If you encounter a site with an extremely high rate of links (a 'crawler trap'), how would you detect and throttle it?
  • How do you ensure the crawler remains distributed and fault-tolerant if a worker node crashes mid-fetch?