Design a recursive distributed file crawler

Design a distributed service that crawls a large file system starting from a root path. A client should be able to call an API to start a crawl job, and background workers should traverse directories and files asynchronously. While processing a directory, a worker may split the work into smaller tasks and enqueue additional async jobs, so the async service can recursively trigger more of its own jobs.

Discuss:

• APIs for creating a crawl job and checking job status
• the data model for crawl jobs, crawl tasks, and discovered file metadata
• how workers recursively schedule child tasks
• how to handle retries, idempotency, deduplication, and partial failures
• how to scale to very large directory trees
• the role of the database, message queue, and async workers
• how to expose progress and final results to clients