Katkat
Back

Design Payment Processing with Idempotency and Reconciliation

RobinhoodSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

Design a resilient payment processing system that interfaces with third-party payment gateways, guarantees exactly-once execution via idempotency, manages complex payment state machines, and resolves discrepancies via reconciliation.

Scenario

Design a core payment processing service for a financial trading platform like Robinhood. The service allows users to initiate deposits and withdrawals (e.g., via ACH or debit rails) by communicating with downstream payment processors and banking partners.

Because network connections can drop, clients may retry requests, and external gateways can fail or delay responses, your system must guarantee that no user is double-charged or credited twice, all funds are accounted for, and in-flight transactions are eventually resolved.

Requirements

Functional Requirements

  • Payment Initiation: Accept payment requests (deposits and withdrawals) with an idempotency key.
  • External Provider Integration: Forward payments to third-party payment rails and capture results (both synchronous responses and asynchronous webhooks).
  • State Management: Accurately track payment states through a well-defined lifecycle (e.g., CREATED, PROCESSING, SUCCEEDED, FAILED, REQUIRES_RECONCILIATION).
  • Reconciliation Engine: Ingest daily clearing/settlement files from partner banks to detect and correct discrepancies between internal records and external reality.

Non-Functional Requirements

  • Strict Idempotency: Duplicate API calls (intentional or accidental retries) must never trigger multiple charges or transfers.
  • Durability & Auditability: Every balance mutation, state change, and API interaction must be immutably recorded for audit compliance.
  • High Availability & Fault Tolerance: Gracefully handle provider downtimes, timeouts, and out-of-order webhooks.

Scale & Numbers

  • Volume: 10 million payment transactions per day (~120 requests/second average, peaking at 600 requests/second).
  • Settlement: Up to 100,000 pending transactions awaiting settlement or asynchronous resolution at any given time.

Key Deliverables

  1. API Design: Endpoints for payment initiation and webhook ingestion.
  2. State Machine & Data Model: Schema for payment records, idempotency keys, and transition states.
  3. Core Architecture: End-to-end flow handling timeouts, retries, and distributed transactions.
  4. Reconciliation Strategy: How to handle mismatches discovered during batch settlement ingestion.

Hints

Hint 1

Enforce a unique database constraint on the client-provided idempotency key paired with a user ID to prevent concurrent duplicate processing at the database level.

Hint 2

Model payment states explicitly with an intermediate state (such as PROCESSING or UNKNOWN) so that network timeouts while calling the gateway do not default to FAILED before verification.

Hint 3

Design the reconciliation engine as an asynchronous batch pipeline that parses end-of-day bank statement files, diffs them against internal ledger entries, and flags deviations for automated resolution or manual review.

Follow-up questions

  • How would you handle a race condition where two identical requests arrive simultaneously at two different web server instances?
  • If the external processor webhook arrives before your synchronous API call completes its database commit, how do you prevent out-of-order state overwrites?
  • Suppose an end-of-day settlement file shows a transaction succeeded, but your service had marked it as failed and already refunded the user. How should the system handle this financial discrepancy?