Design a payment system with real-time holds, batch settlement and safe retries

Design a payment system. The interviewer started with two money movements: placing a hold on the payer's funds in real time when a payment is made, and settling held payments later in batches. The discussion then turned to how retries are handled without charging anyone twice, and from there to consistency and to what happens in specific failure cases. Expect detailed probing in each area.

Constraints and Clarifications

• Who holds the funds was not specified. If the interviewer leaves it open, assume the system authorizes card or bank payments through an external payment processor (the hold) and captures them later in batches (the settlement), and explain how the design changes if the system instead holds users' balances in its own ledger.
• Volumes, settlement frequency and hold expiry rules were not given. Ask, or state your assumptions.

Clarifying Questions

• Is the hold placed through an external processor on a card or bank account, or on a balance the system itself keeps in a wallet ledger?
• What is the peak payment rate, and how many payments go into one settlement batch?
• When does settlement happen: on a fixed schedule such as daily, or when each order is fulfilled? Can a hold be partially captured or released early?
• How long does a hold stay valid before it expires, and what should happen to holds that are never settled?
• Are refunds, chargebacks and multiple currencies in scope?

Part 1 — Real-time hold and batch settlement

Design the flow from a payment request to a hold, and from a hold to its settlement in a batch. Include the components, the payment's lifecycle and the data you store.

What This Part Should Cover

• Components and the synchronous hold path, with its latency target
• A payment state machine covering success, decline, release and expiry
• The data model, including an auditable record of money movements
• How a settlement batch is formed, submitted and confirmed, and how a payment is kept out of two batches

Part 2 — Retries without double charging

Requests are retried by clients, by your own services and by jobs that restart. Explain how the system guarantees that a retried request never holds or settles the same money twice.

What This Part Should Cover

• Idempotency at the API: what is stored, and how concurrent duplicates and changed payloads are handled
• Idempotency toward the processor and in asynchronous workers
• Handling outcomes that are unknown after a timeout
• Which errors are retried, and how

Part 3 — Consistency and failure cases

Explain where the system needs strong consistency and where eventual consistency is acceptable. Then walk through failure cases, such as a crash at the worst possible moment, and how the system recovers from each.

What This Part Should Cover

• What is updated atomically, and how state changes and outgoing events stay in step
• Recovering payments stuck between states
• Reconciliation against the processor's records
• Specific scenarios: a crash after the processor accepted a request, a processor outage, a partially failed batch, an expired hold