Katkat
Back

High Availability and Failover for a Primary-Replica Database

AmazonSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

This problem tests your ability to design a resilient database architecture that minimizes downtime and data loss during node failures.

Scenario

You are tasked with designing a high-availability (HA) layer for a primary-replica PostgreSQL database deployment. Your goal is to ensure the system remains available for read and write operations even if the primary node crashes.

Requirements

  1. Functional: Automate the promotion of a replica to primary if the primary becomes unresponsive. Ensure the system can route client traffic to the new primary.
  2. Non-functional: Minimize Recovery Time Objective (RTO) and Recovery Point Objective (RPO). You must address how to prevent 'split-brain' scenarios where two nodes believe they are the primary simultaneously.
  3. Scale: Assume a deployment spanning two availability zones (AZs).

Discussion Points

Your solution should cover:

  • Health Monitoring: How do you reliably detect a primary node failure versus a transient network partition?
  • Replication Strategy: Briefly explain the choice between synchronous and asynchronous replication and its impact on RPO.
  • Failover Logic: Describe the steps taken during promotion, specifically how you ensure consistency among existing replicas.
  • Traffic Redirection: How do clients discover the new primary after a failover? Describe the role of a service discovery mechanism or a virtual IP.

Hints

Hint 1

Consider using a distributed consensus store like etcd or ZooKeeper to maintain the source of truth for the current primary node identity.

Hint 2

Think about how 'fencing' mechanisms (e.g., STONITH - Shoot The Other Node In The Head) can prevent a partitioned primary from continuing to accept writes.

Hint 3

Evaluate the trade-offs of using a proxy layer versus client-side driver logic for routing traffic to the active database instance.

Follow-up questions

  • How does the choice of synchronous replication affect your system's overall latency and availability?
  • What steps would you take to reintroduce a failed primary node back into the cluster once it recovers?
  • How would you modify this design to support geo-distributed write traffic if latency requirements become extremely strict?