Katkat
Back

Scale a file-crawling API using async jobs

DropboxSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSystem Design

Last updated

This question tests the ability to design an asynchronous distributed system for crawling large file structures, focusing on job orchestration, state management, and scalability under high-latency operations.

System Design: Distributed File Crawler

Design an API and backend system that can crawl the directory structure of large cloud storage accounts (e.g., millions of files). The system must expose an endpoint for a client to trigger a crawl and poll for progress.

Functional Requirements

  • Clients trigger a POST /crawls request, receiving a crawl_id.
  • Clients can poll GET /crawls/{crawl_id} to check status (pending, in-progress, completed, failed) and retrieve partial results.
  • The system must handle millions of files per crawl without timing out.

Non-Functional Requirements

  • Scalability: The system must handle thousands of concurrent crawls.
  • Fault Tolerance: If a worker node crashes, the crawl should resume or retry without losing progress.
  • Throughput: Minimize total crawl time by parallelizing directory traversal.

Constraints

  • The underlying storage API has strict rate limits.
  • The directory tree may contain cycles or extremely deep nesting.

Deliverables

  1. High-level architecture diagram (describe components).
  2. API design (endpoints and status models).
  3. Strategy for managing crawl state and task distribution.
  4. How to handle rate limiting and retries for external storage calls.

Hints

Hint 1

Consider using a message queue or a task distribution system to decouple the API request from the actual file traversal logic.

Hint 2

Think about how to represent the crawl state: would a recursive depth-first search fit in memory, or should you use a breadth-first approach with a distributed task queue?

Hint 3

Address the rate-limiting problem by implementing a global token bucket or a distributed coordinator that tracks requests per user/storage account.

Follow-up questions

  • How would you modify your design to handle a 'stop' command to cancel a crawl mid-execution?
  • What changes would be necessary to ensure that if a crawl is triggered twice for the same account, the work is deduplicated or shared?
  • How would you handle monitoring and observability to detect stalled crawl jobs or hot spots in the queue?