Katkat
Back

Prevent Duplicate Payments Under High Load

RipplingSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

Design a highly reliable payment processing system that guarantees exactly-once execution and prevents duplicate charges during high-concurrency spikes and downstream failure scenarios.

Scenario

At a workforce management company like Rippling, companies run payroll and contractor payouts on fixed schedules (such as the 1st and 15th of each month). During these peak windows, millions of disbursements are triggered simultaneously via scheduled batch workers, API integrations, and manual employer retries.

Downstream payment gateways and banking rails (e.g., ACH, FedNow, Stripe) often experience high latency or intermittent connection drops. The system must guarantee that no employee or contractor is ever paid twice, even if clients aggressively retry timed-out requests or duplicate worker threads process the same payload.

Requirements

Functional Requirements

  • Payment Initiation: Accept payment requests containing an idempotency key, source account, destination account, amount, and currency.
  • Idempotency Guarantee: If a request with the same idempotency key is submitted multiple times, the system must return the exact same response as the initial successful or in-progress call without executing a duplicate transfer.
  • Status Querying: Provide an endpoint to check the real-time status of any transaction.
  • State Reconciliation: Handle "in-doubt" payment states resulting from downstream gateway timeouts or network partitions.

Non-Functional Requirements

  • Correctness: Zero duplicate payments under any failure or concurrency scenario.
  • High Availability: 99.99% availability for payment submission; 99.999% for status lookups.
  • Latency: Client API response acknowledgment in under 250 ms (asynchronous downstream execution is acceptable).

Scale & Traffic Assumptions

  • Peak Volume: 5,000 payment requests per second during peak payroll cutoff windows.
  • Daily Volume: 20 million disbursements on peak payroll days.
  • Downstream Latency: Banking networks take between 500 ms and 5,000 ms to settle an individual payment transaction.

What Your Design Should Cover

  1. API & Idempotency Key Design: Define the request/response payloads, key generation strategy, and caching/storage mechanics for idempotency records.
  2. Data Model & State Machine: Design the payment state machine (e.g., CREATED, PENDING_EXECUTION, SENT_TO_BANK, SETTLED, FAILED, INDETERMINATE) and relational/transactional schema.
  3. Concurrency Control: Explain how you prevent race conditions when two identical requests hit different backend nodes at the exact same millisecond (e.g., distributed locks vs. unique database constraints).
  4. Failure Modes & Gateway Integration: Detail how you handle downstream HTTP 504 timeouts, reverse lookups, reconciliation jobs, and ledger consistency.

Hints

Hint 1

Consider enforcing idempotency at the database layer using unique constraints on the idempotency key rather than relying solely on distributed cache locks.

Hint 2

Model payment lifecycles as a strict, non-reversible finite state machine to prevent race conditions during async status updates.

Hint 3

For downstream timeouts where the external gateway state is unknown, avoid immediate failure or retry; mark the state as 'in-doubt' and use automated polling or out-of-band reconciliation.

Follow-up questions

  • How do you handle a scenario where two identical requests arrive at the same millisecond with matching idempotency keys, but conflicting request payloads (e.g., different amounts)?
  • How does your design scale the idempotency data store over multi-year retention requirements without degrading peak database write performance?
  • If the external bank gateway does not natively support idempotency keys, how do you prevent duplicate execution when retrying a timed-out call?