Make a Charge-Then-Record Payment Endpoint Safe Against Timeouts, Retries and Crashes

A payment endpoint in a web application works in two steps. It first calls an external payment provider to charge the customer, and then writes a record of the payment to the application's own database. You are asked what is wrong with this design and how to fix it.

The flow, reconstructed as a sketch:

def pay(order_id, user_id, amount):
 charge = provider.charge(user_id, amount) # external HTTP call
 db.insert_payment(order_id, user_id, amount, charge.id, status="paid")
 return {"status": "paid"}

Assume the provider accepts an idempotency key on charge requests, lets you look up the status of a charge, and can notify you of charge results through webhooks. Confirm what the real provider offers before relying on any of these.

Clarifying Questions

• Who retries a failed or timed-out request: the browser or mobile client, this server, or both?
• Can the provider report a charge as pending, or does every charge call end in success or a decline?
• For how long does the provider remember an idempotency key, and what does it return when it sees the same key again?
• Is each order paid by exactly one payment, so that an order must never be charged twice?

Part 1 — What can go wrong

List the failure modes of the code above. For each one, say what state the customer's money and your database end up in.

What This Part Should Cover

• Failures between the external charge and the local write, including the case where the outcome of the charge is unknown
• How retries and concurrent submissions of the same payment behave
• Whether a local database transaction can cover the external charge

Part 2 — Make retries safe

Redesign the flow so that a client or server retry of the same payment never produces a second charge or a second record.

What This Part Should Cover

• Where the identity of a payment attempt comes from and how long it lives
• How the server detects a duplicate request, including two duplicates that arrive at the same moment
• How the call to the provider is itself protected against duplicates

Part 3 — Never lose track of a charge

Some payments will end with an unknown outcome: the process crashed mid-flow, or the provider call timed out. Change the flow so that no charge attempt can be lost track of, and explain how each unknown outcome is eventually resolved.

What This Part Should Cover

• The states of a payment record and the allowed transitions between them
• Every path by which an unknown outcome gets resolved, and who owns each path
• Handling the same result arriving more than once, or results arriving in an unexpected order

Part 4 — Where the lock goes

Duplicate requests for the same payment, a webhook, and a background recovery process may all try to update the same record. Where would you put a lock, and should it be held until the external provider call finishes?

What This Part Should Cover

• What actually needs mutual exclusion, and at what granularity
• The cost of holding a lock or database transaction across a network call
• Ways to get the same safety without a long-held lock