Make a Courier Pay Workflow Resilient to Downstream Failures

Problem Statement

In a courier platform, couriers must be paid accurately once deliveries are completed. You are tasked with implementing the core execution engine of a PayoutProcessor that processes payment instructions and posts them to an external, unreliable downstream ledger service.

The downstream ledger service has the following characteristics:

• It accepts an idempotency_key, courier_id, and amount_cents.
• It can return three types of responses:
  • SUCCESS: Payout was recorded.
  • TRANSIENT_ERROR: Server busy or network timeout (unknown whether the ledger processed the request or not).
  • TERMINAL_ERROR: Permanent failure (e.g., invalid courier account or closed ledger account). Retrying will never succeed.
• It exposes a check_status(idempotency_key) endpoint that returns SUCCESS, NOT_FOUND, or FAILED.

Your task is to implement the PayoutProcessor class with state persistence to guarantee:

1. No Duplicate Payouts: A courier is never paid more than once for a given payout instruction.
2. Durable Intent: Payout states are tracked (PENDING, SUBMITTED, COMPLETED, FAILED_TERMINAL, REQUIRES_RECONCILIATION).
3. Ambiguous Timeout Handling: When a timeout or transient error occurs, the processor must probe the downstream service using check_status before attempting a retry or marking the job.
4. Bounded Retries: If downstream remains unreachable after a configurable max_retries, mark the payout as REQUIRES_RECONCILIATION so automated background reconciliation or an on-call engineer can inspect it safely.

Interface

from enum import Enum
from typing import Optional

class PayoutStatus(Enum):
    PENDING = "PENDING"
    COMPLETED = "COMPLETED"
    FAILED_TERMINAL = "FAILED_TERMINAL"
    REQUIRES_RECONCILIATION = "REQUIRES_RECONCILIATION"

class LedgerResponse(Enum):
    SUCCESS = "SUCCESS"
    TRANSIENT_ERROR = "TRANSIENT_ERROR"
    TERMINAL_ERROR = "TERMINAL_ERROR"

class LedgerStatus(Enum):
    SUCCESS = "SUCCESS"
    NOT_FOUND = "NOT_FOUND"
    FAILED = "FAILED"

class PayoutProcessor:
    def __init__(self, ledger_client, store, max_retries: int = 3):
        """
        ledger_client methods:
          - post_payment(idempotency_key: str, courier_id: str, amount_cents: int) -> LedgerResponse
          - check_status(idempotency_key: str) -> LedgerStatus
        
        store methods:
          - get_payout(payout_id: str) -> Optional[dict]
          - save_payout(payout_id: str, data: dict) -> None
        """
        pass

    def process_payout(self, payout_id: str, courier_id: str, amount_cents: int) -> PayoutStatus:
        """
        Executes the payout safely and returns the terminal or current state.
        """
        pass

Constraints

• amount_cents > 0
• payout_id is globally unique per delivery payout intent.
• Downstream idempotency_key must be deterministically derived from payout_id (e.g., f"pay_{payout_id}").
• store.save_payout writes synchronously to a persistent database.
• The function should not loop indefinitely; retries must not exceed max_retries.

Example

# Scenario: Transient network drop on first attempt, succeeded on check_status
# 1. store.save_payout(payout_id, status=PENDING)
# 2. ledger_client.post_payment(...) -> LedgerResponse.TRANSIENT_ERROR
# 3. ledger_client.check_status(...) -> LedgerStatus.SUCCESS
# 4. store.save_payout(payout_id, status=COMPLETED)
# Result: PayoutStatus.COMPLETED