Katkat
Back

Design a Smooth Per-Customer Calls-Per-Second Controller

Retell AISoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSystem Design

Last updated

Design a distributed outbound call pacing service that enforces smooth per-tenant calls-per-second (CPS) rate limits while ensuring durable queueing, priority scheduling, and safe failover.

Scenario

Retell AI enables businesses to initiate conversational AI voice agents for automated outbound calls (e.g., appointment reminders, customer support callbacks, outreach). Downstream telephony carriers (such as Twilio or direct SIP trunk providers) enforce strict, per-customer calls-per-second (CPS) limits. Bursting calls within a 1-second window can lead to carrier-level throttling (SIP 503 or HTTP 429), carrier carrier-penalties, or degraded voice latency.

Design a multi-tenant call pacing engine that accepts call requests via an API and smoothly dispatches them to telephony carriers strictly obeying each tenant's configured CPS limit.

Requirements

Functional Requirements

  • Durable Enqueueing: Customers can enqueue outbound call requests with metadata (destination number, agent ID, priority level).
  • Smooth Pacing: Dispatch calls with uniform inter-call delays (interval = 1 / CPS) rather than bursting at the boundary of each second.
  • Priority & Fairness: Support tiered priorities (e.g., urgent transactional alerts over batch marketing campaigns) without starving lower-priority calls indefinitely.
  • Retries & Backpressure: Handle carrier-level rate-limiting or temporary outages with exponential backoff and propagate backpressure when tenant queues grow unbounded.
  • Observable Semantics: Provide real-time status visibility (enqueued, dispatched, failed, dropped) for observability.

Non-Functional Requirements

  • Tenant Isolation: A surge in call volume from one customer must not delay or degrade another customer's dispatch rate.
  • High Availability & Fault Tolerance: If a pacing worker crashes, call dispatching must resume seamlessly with no double-calling (at-most-once execution for call initiation).
  • Precision & Low Jitter: Dispatch timing jitter should be kept under 50 milliseconds.

Scale & Performance Assumptions

  • Tenants: 10,000 active enterprise tenants.
  • Tenant Limits: Vary from 1 CPS up to 500 CPS per tenant.
  • Aggregate Volume: Average 10,000 CPS total; peak load up to 50,000 CPS aggregate across the platform.
  • Durability: Zero dropped calls once an enqueue request is acknowledged (202 Accepted).

Key Deliverables

  1. API & Data Model: Endpoints for call submission and tenant CPS configuration, plus storage schemas for call states.
  2. Architecture & Pacing Algorithm: Core system components (ingestion, durable queues, pacing engine, dispatcher) and algorithm choice (e.g., Leaky Bucket vs. Virtual Clock/Calendar Queues).
  3. Partitioning & Failover: How tenants and calls are assigned to workers, leader leases, fencing tokens, and split-brain prevention.
  4. Failure Modes & Trade-offs: Recovery from worker crashes, clock skew mitigation, and downstream carrier backpressure handling.

Hints

Hint 1

A standard token bucket can cause a 1-second burst; consider a leaky bucket or virtual-time scheduler to ensure consistent inter-dispatch intervals.

Hint 2

Hash tenants across partitions to maintain tenant isolation, but use consistent hashing with virtual nodes to handle uneven CPS configurations across tenants.

Hint 3

Use epoch-based fencing tokens or distributed leases (e.g., via etcd or Redis Redlock with monotonically increasing versions) to prevent duplicate dial attempts during worker failover.

Follow-up questions

  • How would you dynamically adjust a customer's pacing rate if the downstream carrier returns SIP 503 Service Unavailable or HTTP 429 despite operating within configured CPS?
  • If a customer with 50 CPS enqueues 2,000 low-priority calls and suddenly enqueues 10 high-priority calls, how does your scheduling queue minimize latency for the high-priority calls without starving the queue?
  • How would you design the system to handle cross-region active-active deployments where a customer might submit calls to both US-East and US-West?