How would you throttle a crawler?
NooksSoftware Engineer · OnsiteLast updated
This question evaluates competency in scalable distributed system design, including rate limiting, request scheduling, concurrency control, fault tolerance, backoff/retry strategies, deduplication, and observability for networked crawlers.
The crawler above is now a production service. Expanding each page requires an outbound HTTP or API call to fetch its links, and upstream services may enforce strict rate limits.
Design a crawler that:
- Traverses a large set of pages or resources efficiently
- Avoids per-host bottlenecks
- Does not overwhelm upstream services or accidentally cause a denial-of-service
- Handles retries, timeouts, and partial failures safely
- Scales across multiple workers or machines
Discuss the architecture, concurrency model, request scheduling, deduplication, rate limiting, backoff strategy, fairness across domains, and observability.
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.