Katkat
Back

Design a cloud database write path and recovery

AmazonSoftware Engineer · Technical Screen
HardSoftware EngineerTechnical ScreenSystem Design

Last updated

This question tests the ability to architect a distributed database write path that decouples compute from storage, specifically focusing on how to maintain ACID properties during failures.

Problem Statement

Design a cloud-native relational database architecture that separates the compute layer (query processing/transaction management) from the storage layer (distributed, log-structured storage). The system must ensure high durability and availability while handling compute node failures.

Requirements

  1. Write Path: Describe the sequence of operations from when a client sends an INSERT/UPDATE until the database acknowledges the commit.
  2. Durability: Explain how the system guarantees durability if a compute node crashes immediately after receiving a write but before flushing to data pages.
  3. Recovery: Outline the process for a new compute node to take over the state of a failed node. How does it reconstruct the consistent state of the database?
  4. Replication: Define the quorum mechanism for the storage nodes. How do you handle storage node latency or network partitions without stalling the write path?

Assumptions

  • The storage layer consists of multiple independent storage nodes that store log fragments.
  • The compute layer is stateless regarding the actual data pages, but manages the transaction log.
  • You must address how to minimize write amplification and how to handle the "slowest-disk" problem in your quorum.

Expected Output

  • A high-level architectural diagram or component description.
  • The specific protocol used for log replication (e.g., quorum-based or leader-based).
  • Discussion on how to reconcile the Log Sequence Number (LSN) during recovery.
  • Trade-offs between latency and strict consistency.

Hints

Hint 1

Consider the difference between flushing data pages to disk versus appending to a distributed write-ahead log (WAL).

Hint 2

Think about using a Quorum consensus algorithm (like Paxos or Raft) for the log segments rather than the full data pages.

Hint 3

Address the 'Read-After-Write' consistency model and how a new compute node fetches the tail of the log during recovery.

Follow-up questions

  • How would you optimize the write path if the storage layer becomes the bottleneck during peak write bursts?
  • If you are using a distributed WAL, how do you perform garbage collection on the log segments that have already been persisted to data pages?
  • How would you incorporate observability to detect if a specific storage node is lagging in its LSN acknowledgments?