Katkat
Back

Design a Payment Orchestration System

OpenAISoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

Design a resilient merchant-side payment orchestration platform that routes transactions across multiple payment service providers (PSPs) while enforcing strict amount invariants, idempotent state transitions, and auditability.

Scenario

You are tasked with designing a core Payment Orchestration System for a high-volume global digital merchant (e.g., handling subscriptions and one-off purchases). The platform acts as an abstraction layer between internal checkout services and multiple external Payment Service Providers (PSPs such as Stripe, Adyen, and PayPal).

The system must orchestrate the complete lifecycle of payment transactions—authorization, capture, void, and refund—while dynamically routing across providers, managing transient network failures, and ensuring that no customer is ever double-charged or over-refunded.

Requirements

Functional Requirements

  • Payment Lifecycle Operations: Expose APIs to authorize, capture, void, and refund payments.
  • Dynamic Provider Routing & Failover: Support rule-based provider selection with automated failover when a primary provider experiences degraded health or network timeouts.
  • Idempotency & Concurrency: Ensure exactly-once execution semantics for client requests, preventing duplicate charges or invalid state transitions during concurrent retries.
  • Asynchronous Webhook Processing: Ingest and process external PSP webhooks that may arrive out of order, delayed, or duplicated.
  • Reconciliation & Auditing: Maintain an immutable audit log and double-entry ledger representation to verify that refunded_amount <= captured_amount <= authorized_amount at all times.

Non-Functional Requirements

  • Consistency: Strong consistency for payment state transitions and balance invariants.
  • Availability: 99.999% availability for the authorization ingestion path.
  • Latency: p99 latency under 500 ms (excluding downstream third-party processor latency).
  • Security & Compliance: Avoid storing raw primary account numbers (PAN) directly by integrating tokenization gateways (PCI-DSS Level 1 scope reduction).

Scale Assumptions

  • Volume: 50 million transactions per day (~600 transactions/sec average, 3,500 transactions/sec peak).
  • Webhook Ingestion: Up to 10,000 events/sec peak from third-party PSPs.
  • Storage Retention: 7-year retention for financial records and audit trails.

Key Deliverables

  1. High-Level Architecture: End-to-end component diagram from client checkout to PSP execution.
  2. State Machine & Ledger Model: Entities, transaction states, and schema enforcing monetary invariants.
  3. Failure Handling & Failover: Handling PSP network timeouts without double authorizations.
  4. Webhook & Out-of-Order Ingestion: Architecture for reconciling webhook events against active transactional states.

Hints

Hint 1

Differentiate between business idempotency keys from the checkout client and downstream provider transaction IDs to prevent duplicate calls across multiple PSP attempts.

Hint 2

Use a deterministic finite state machine (FSM) backed by an immutable ledger or event store to guarantee that transitions like capture or refund strictly depend on verified preceding states.

Hint 3

When a PSP request times out, treat the transaction status as unknown rather than failed; issue an explicit status check or reverse transaction before attempting failover to a secondary provider.

Follow-up questions

  • How does your system handle a network partition where a request to authorize times out at the PSP, and the client retries with the same idempotency key 100ms later?
  • How do you design the ledger schema to support partial refunds across multi-item orders without running into race conditions during concurrent refund requests?
  • If a PSP delivers an out-of-order webhook marked 'CAPTURED' before the synchronous response to your initial 'AUTHORIZE' call has been persisted, how does your system resolve the state?