Katkat
Back

Design a Hierarchical File System

DatabricksSoftware Engineer · Onsite
HardSoftware EngineerOnsiteSystem Design

Last updated

This question tests the ability to design a distributed file system namespace capable of handling millions of files and directories with strong consistency and high availability.

Design a Hierarchical File System

Design a distributed, hierarchical file system (similar to HDFS or GCS) that supports a namespace of billions of files and directories. The system must support concurrent reads and writes, handle high throughput, and ensure metadata consistency.

Requirements

  1. Functional Requirements:

    • Operations: create(path), delete(path), list(path), read(path), write(path, content), and rename(old_path, new_path).
    • Hierarchical structure: Folders contain files and other folders.

  2. Non-functional Requirements:

    • Scalability: The system must scale to billions of files.
    • Consistency: Changes to the namespace must be linearizable.
    • High Availability: The metadata layer must survive individual node failures.

Expectations

  • Provide a high-level architecture diagram or description.
  • Explain how you handle the separation of metadata (namespace, permissions) and data (actual file chunks).
  • Detail how metadata is partitioned and replicated to avoid a single point of bottleneck/failure.
  • Discuss how rename operations (which involve changing tree nodes) are handled in a distributed environment.
  • Address how the system handles metadata consistency during concurrent modifications.

Hints

Hint 1

Consider a central Metadata Server (NameNode) model vs. a distributed metadata model using consistent hashing or partitioned trees.

Hint 2

Think about how to use distributed consensus algorithms like Raft or Paxos to manage metadata replicas and maintain consistency.

Hint 3

Address the 'Rename' operation carefully; updating a directory tree structure atomically is significantly harder than creating a leaf node.

Follow-up questions

  • How would you implement file permissions (POSIX-like) and ensure they are checked efficiently without querying a central authority for every read?
  • If the metadata grows too large for the memory of a single cluster, how would you re-architect the system to shard the namespace?
  • What specific mechanisms would you use to garbage collect orphaned blocks that were written but never committed to the namespace?