Katkat
Back

Design an Idempotent Payment Processing Platform

SalesforceMember of Technical Staff · Onsite
MediumMember of Technical StaffOnsiteSystem Design

Last updated

Design a resilient payment platform that guarantees zero duplicate charges and safely resolves ambiguous downstream gateway timeouts.

Scenario

You are tasked with designing the core payment execution platform for an enterprise cloud ecosystem (such as Salesforce Billing). The platform accepts payment requests from internal billing pipelines and customer-facing checkouts, orchestrates transactions with external payment gateways (e.g., Stripe, Adyen), and maintains an immutable financial ledger.

The primary engineering challenge is ensuring strict idempotency: network drops, client retries, asynchronous webhook delivery, and downstream provider timeouts must never result in double-charging a customer or creating inconsistent ledger states.

Requirements

Functional Requirements

  • Idempotent Charge Execution: Accept payment requests with an idempotency key. Repeating a request with the same key and payload must return the original result without re-executing the charge.
  • Ambiguous Timeout Handling: If an external gateway call times out or returns an HTTP 5xx, safely determine whether the charge succeeded, failed, or remains in-flight.
  • Out-of-Order Webhook Ingestion: Ingest asynchronous gateway webhooks concurrently with synchronous client requests without corrupting state.
  • Reconciliation & Repair: Provide automated background reconciliation against gateway transaction logs and support guarded manual operator interventions for stuck transactions.
  • Audit Ledger: Maintain an append-only, tamper-evident audit history of all state changes.

Non-Functional Requirements

  • Correctness Over Latency: Zero duplicate charges under any failure mode, including network partitions and database failovers.
  • Availability: 99.99% uptime for payment ingestion.
  • Scale: 100 million transactions per day (average ~1,200 TPS, peak ~5,000 TPS).
  • Precision: Exact monetary calculations without floating-point inaccuracies, supporting multi-currency transactions.

What Your Design Should Cover

  1. API & Data Model: Idempotency key lifecycle, request fingerprinting, payment transaction states, and ledger schema.
  2. State Machine & Concurrency Control: Transition flow (e.g., INITIATED -> GATEWAY_IN_FLIGHT -> SUCCEEDED / FAILED / INDETERMINATE) and how database locks or distributed coordination prevent concurrent duplicates.
  3. Gateway Integration Architecture: How to isolate synchronous HTTP timeouts, manage bounded retries with exponential backoff, and poll gateway status endpoints.
  4. Webhook & Event Ordering: Mitigating race conditions between webhook callbacks and synchronous execution.
  5. Reconciliation & Operator Overrides: Batch verification against daily bank/gateway Settlement Reports and safeguards around manual repairs.

Hints

Hint 1

Enforce a database-level unique constraint on `(tenant_id, idempotency_key)` alongside a cryptographic hash of the request payload to catch concurrent calls and payload mismatches.

Hint 2

Model payment lifecycles as a strict, monotonic state machine where terminal states cannot be overwritten by out-of-order webhooks or late-arriving retry responses.

Hint 3

Treat downstream gateway network timeouts as an indeterminate state rather than an immediate failure, routing them to a dedicated verification worker before releasing or confirming funds.

Follow-up questions

  • How does your system respond if a client sends an existing idempotency key with a mutated payload (e.g., a different amount or currency)?
  • How do you prevent a distributed race condition if an asynchronous webhook arrives and tries to update the transaction before the synchronous gateway call has returned?
  • If an active-active multi-region deployment is required for high availability, how do you prevent duplicate execution during a regional network partition?