Katkat
Back

Design a Kafka-Based File Watcher for Millions of Files

AmazonSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSystem Design

Last updated

Design a Kafka file watcher for millions of files with source sharding, durable events, per-file ordering, missed-event reconciliation, and version-aware reads.

Design a file-watcher system that handles millions of files and uses Kafka to deliver change events to downstream consumers. Explain initial discovery, ongoing changes, ordering, recovery, and how the system detects changes missed by native filesystem notifications.

Constraints & Assumptions

  • Clarify whether files are on local disks, many hosts, or a shared filesystem, and what event guarantees those sources provide.
  • Define the desired result: every low-level event, or eventual convergence to the current file state. Those are different guarantees.
  • The source specifies scale in file count but no event rate, file size, or latency target.

Clarifying Questions to Ask

  • Which changes matter: creation, content updates, deletion, rename, or metadata changes?
  • Is there a durable source journal, or can watch events overflow or disappear during downtime?
  • Is ordering required per file, per directory, per host, or globally?
  • Do consumers need file contents or only metadata and a versioned reference?

Hints

Hint 1

Reconcile after a gap: A durable Kafka log cannot recover a filesystem event that was lost before it reached the producer. Define how source state is checked when notification continuity is uncertain.

Follow-up questions

  • How would an event overflow during a burst of changes be detected and repaired?
  • What happens if a file changes again before a consumer reads its contents?