Design a Reliable Scheduler for Payment Jobs

Scenario

At a hospitality marketplace like Airbnb, many payments do not occur immediately at booking time. For example, guests may pay a split balance two weeks before check-in, or host payouts may be released 24 hours after a guest checks in.

Design a reliable, distributed scheduler responsible for executing these scheduled, one-time payment jobs. The system must guarantee that scheduled jobs run close to their target execution time, never result in duplicate charges or payouts, and handle worker crashes, network partitions, and cancellation races gracefully.

Requirements

Functional Requirements

• Schedule a Job: Support scheduling payment events with a specific execution timestamp (execute_at).
• Execution & Retries: Ensure each scheduled job executes reliably at or shortly after its target time. Retry failed jobs with exponential backoff for transient errors.
• Cancellation: Allow clients to cancel a scheduled payment (e.g., booking cancelled before balance is charged).
• Idempotency & Exactly-Once Semantics: Prevent duplicate charges at both the scheduler layer and downstream payment gateways.
• Reconciliation: Detect and resolve stale leases, stuck jobs, and state discrepancies between the scheduler, internal ledger, and external payment processors.

Non-Functional Requirements

• High Reliability & Durability: Zero lost jobs and strict guarantees against double charges.
• Timing Accuracy: Jobs should be claimed within 5–10 seconds of their scheduled time.
• Availability: High availability for scheduling and cancellation APIs (99.99%), with strong consistency for execution state transitions.

Scale Assumptions

• Total active scheduled jobs: ~50 million at any given time.
• Daily executions: ~5 million jobs executed per day.
• Peak throughput: Up to 3,000 executions per second.

Key Areas to Cover

1. Data Model & Storage: Database selection and schema design for tracking job state, lease tokens, execution attempts, and payload metadata.
2. Job Dispatch & Lease Claiming: How workers poll or receive jobs to execute, including distributed locking/lease mechanisms to prevent multiple workers from claiming the same job.
3. Cancellation & Race Conditions: Managing races where a cancel request arrives concurrently with worker claiming or payment gateway dispatch.
4. Provider Idempotency: Safe interaction with third-party payment gateways during network timeouts and retries.
5. Recovery & Reconciliation: Detecting hung workers, expired leases, and reconciling state against payment gateway logs.