Design Distributed Storage for Large Structured Data

Design a distributed filesystem or storage service for large structured datasets, similar in purpose to a distributed data-processing filesystem. Go beyond a component diagram: explain concrete flows for metadata and data access, caching at different layers, job queues, node failures, and load balancing.

Constraints & Assumptions

• Clarify whether files are immutable, append-only, or randomly mutable, and how compute jobs consume them.
• Separate namespace metadata from large data blocks.
• No node count, data size, or latency target is supplied. State design choices without inventing deployment facts.

Clarifying Questions to Ask

• Is the workload dominated by large sequential scans, appends, or small random reads?
• What durability and availability are required when a node or rack fails?
• Are processing jobs part of this service or clients of it, and how important is data locality?
• Which cached data may be stale, and how is a file version identified?

Part 1 — Storage and cache flows

Describe writing and reading a large file, including block placement, metadata publication, and cache use.

What This Part Should Cover

• Block IDs, file versions/manifests, metadata authority, and durable write acknowledgement.
• Client metadata caches, block/data caches, and their invalidation or versioning rules.
• Locality and bounded read amplification for structured data.

Part 2 — Node failure and recovery

Describe detection, replica selection, repair, and metadata availability.

What This Part Should Cover

• Checksums, failure detection, placement across failure domains, and repair prioritization.
• A clear authoritative metadata state under failover.
• Recovery from interrupted writes without publishing incomplete files.

Part 3 — Jobs and load balancing

Explain how queued work is assigned and how storage/compute resources are balanced.

What This Part Should Cover

• Job identity, attempts, retries, locality-aware scheduling, and output commit rules.
• Capacity-aware placement and rebalancing with bandwidth limits.
• Backpressure and metrics separating metadata, disk, network, and worker bottlenecks.