Katkat
Back

Design a Distributed Rate Limiter

OpenAISoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSystem Design

Last updated

Tests your ability to architect a scalable, low-latency distributed rate limiter capable of enforcing multi-dimensional limits (such as requests per minute and tokens per minute) across distributed services.

Product Scenario

You are tasked with designing a centralized, distributed rate-limiting service for an AI API platform (similar to OpenAI). The platform serves millions of developers who make inference requests against large language models. The rate limiter must guard upstream compute clusters against abuse, ensure fair resource sharing, and enforce tier-based quotas.

Requirements

Functional Requirements

  • Multi-dimensional rate limiting: Support limits based on both Requests Per Minute (RPM) and Tokens Per Minute (TPM) per API key or organization.
  • Dynamic rules & tiers: Allow configurable quotas per user tier (e.g., Free, Tier 1, Tier 5) and per model endpoint (e.g., gpt-4o vs embedding).
  • Client feedback: Return HTTP 429 Too Many Requests when limits are exceeded, along with standard rate-limit headers (Retry-After, X-RateLimit-Remaining, X-RateLimit-Reset).
  • Post-request reconciliation: Support updating token counts accurately after a request completes, since total tokens consumed are only known after response generation.

Non-Functional Requirements

  • Ultra-low latency: The rate-limiting check must add negligible overhead to the request path (target: p99 < 5ms).
  • High availability: The rate limiter must not become a single point of failure. Failure of the rate-limiting infrastructure should degrade gracefully.
  • Scalability: Must handle 200,000+ incoming requests per second across globally distributed gateway instances.
  • Configurable consistency: Balance precision against latency—slight over-allowance during network partitions is preferable to blocking valid traffic.

Scale & Traffic Assumptions

  • Peak incoming throughput: 200,000 requests/sec.
  • Active API keys: 10 million.
  • Average token payload: 1,500 tokens per request.

What Your Answer Should Cover

  1. Rate Limiting Algorithm: Evaluate Token Bucket, Leaky Bucket, and Sliding Window Counter; justify your choice for this use case.
  2. High-Level Architecture: Interaction between API Gateways, rate-limiter components, caching layers, and the persistent configuration store.
  3. Data Model & Concurrency: Key-value schema and atomic update mechanisms (e.g., Redis with Lua scripts, distributed locks, or memory-mapped counters).
  4. Handling Post-Execution Costs: Architecture for reserving estimated tokens upfront and reconciling actual token usage post-generation.
  5. Failure Modes & Edge Cases: Fallback behavior during cache failures, network splits, clock drift, and cold starts.

Hints

Hint 1

Compare Token Bucket and Sliding Window Counter in terms of memory footprint, atomicity of updates, and ability to handle bursty token reservations.

Hint 2

To achieve sub-5ms latency at scale, consider a hybrid approach with local in-memory batching or token leases on the edge gateway rather than a remote call on every request.

Hint 3

Because LLM responses are streamed and final token counts are dynamic, design a two-phase check: reserve an estimated quota before execution, then reconcile actual usage upon completion.

Follow-up questions

  • How would you handle a distributed denial-of-service (DDoS) attack targeting invalid or randomized API keys without overloading the rate-limiter storage?
  • If you expand to multi-region deployments, how do you prevent split-brain quota exhaustion without adding cross-region network latency to the critical path?
  • How would you implement 'bursting' allowances (e.g., allowing 2x limit for up to 10 seconds) without violating overall capacity constraints?