Build a Three-Service Refund Workflow over HTTP

Scenario

You are tasked with building a distributed refund workflow for an order fulfillment platform. The system consists of three independent HTTP services running on separate ports:

1. Order Service (:8001): Manages order records and tracks cumulative refunded amounts.
2. Payment Service (:8002): Handles communication with third-party payment rails to disburse funds.
3. Refund Orchestrator (:8003): Executes a durable multi-step refund workflow using a directed execution graph.

All monetary values must be processed as non-negative integer cents (e.g., $10.50 = 1050).

Service Contracts

1. Order Service (:8001)

• GET /orders/{order_id}
  • Response 200: {"order_id": string, "total_amount_cents": int, "refunded_amount_cents": int, "status": "COMPLETED" | "CANCELLED"}
• POST /orders/{order_id}/holds
  • Body: {"hold_id": string, "amount_cents": int}
  • Validates refunded_amount_cents + amount_cents <= total_amount_cents.
  • Response 200: {"hold_id": string, "status": "HELD"}
  • Response 409: Conflict if concurrent requests exceed remaining balance.
• POST /orders/{order_id}/commit-hold
  • Body: {"hold_id": string}
  • Marks the held amount as permanently refunded.

2. Payment Gateway Service (:8002)

• POST /payments/refund
  • Body: {"idempotency_key": string, "order_id": string, "amount_cents": int}
  • Simulates payment gateway execution. If invoked with an existing idempotency_key, returns the recorded status without re-executing.
  • May occasionally respond with HTTP 504 Gateway Timeout or connection drops.

3. Refund Orchestrator (:8003)

• POST /refunds
  • Body: {"refund_id": string, "order_id": string, "amount_cents": int}
  • Executes the workflow steps: VALIDATE -> HOLD -> DISBURSE_PAYMENT -> COMMIT_ORDER.
  • Response 202: {"refund_id": string, "status": "PROCESSING"}
• GET /refunds/{refund_id}
  • Response 200: Returns workflow state, step statuses, and final result (SUCCESS or FAILED).

Requirements

1. Idempotency & Concurrency: Two concurrent requests attempting to refund an order must not cause refunded_amount_cents to exceed total_amount_cents. Repeated calls with the same refund_id must return the existing workflow progress.
2. Crash & Restart Recovery: The orchestrator must persist step progress to disk (e.g., SQLite or a JSON write-ahead log). If the orchestrator process crashes mid-workflow and restarts, it must resume incomplete steps rather than starting over.
3. Ambiguity Resolution: When the payment gateway times out (504), the orchestrator must poll or replay using the exact same idempotency_key until a definitive outcome is determined.

Example

// POST :8003/refunds
{
  "refund_id": "ref_123",
  "order_id": "ord_999",
  "amount_cents": 1500
}

// Step 1: GET :8001/orders/ord_999 -> Balance available: 2000 >= 1500
// Step 2: POST :8001/orders/ord_999/holds -> Hold "hold_ref_123" granted
// Step 3: POST :8002/payments/refund -> Payment settled
// Step 4: POST :8001/orders/ord_999/commit-hold -> Committed

// GET :8003/refunds/ref_123
{
  "refund_id": "ref_123",
  "status": "SUCCESS",
  "steps": {
    "hold": "COMPLETED",
    "payment": "COMPLETED",
    "commit": "COMPLETED"
  }
}