Design a Billing-Token Rate Limiter for AI Agents

Scenario

AI agent platforms execute long-running, multi-step tasks using Large Language Models (LLMs). Unlike traditional web APIs where each HTTP call consumes one unit of quota, LLM calls consume a variable number of input and output "billing tokens" (e.g., 500 prompt tokens + 1,200 generated completion tokens). Because completion length is unknown before inference finishes, traditional request-rate limiters cannot prevent overages or manage customer token budgets effectively.

Design a distributed token-rate limiter and quota management system that enforces per-minute, per-hour, and monthly billing-token limits across multiple AI agents belonging to an organization.

Requirements

Functional Requirements

• Reserve-and-Commit Workflow: Support reserving an estimated number of tokens prior to LLM inference, followed by committing the actual tokens consumed (or rolling back unused reserved tokens).
• Hierarchical Scopes: Enforce limits across multiple levels: per-agent, per-user, and organization-wide pools.
• Configurable Throttling Policies: Support both hard limits (reject request immediately with HTTP 429) and soft limits / queuing (allow prioritized burst or delay execution).
• Streaming Support: Handle token reconciliation for streaming LLM responses that output tokens over several seconds or minutes.

Non-Functional Requirements

• Low Latency Overhead: Reservation checks must complete in under 5 ms at p99 to avoid stalling generation pipelines.
• High Availability & Fault Tolerance: Failure of the rate limiter must not silently drop generation results; define a clear fail-open or fail-closed strategy.
• Accuracy vs. Throughput: Balance strict consistency against network latency for high-frequency callers.
• Observability: Emit real-time metrics on quota saturation, rejection counts, and token burn rate for billing pipelines.

Scale Assumptions

• 100,000 active organizations.
• 50,000 reservation/commit operations per second at peak.
• Mean reservation duration (time between reserve and commit): 4 seconds.
• 90% of requests are non-streaming; 10% are streaming responses that last up to 60 seconds.

What to Cover

1. API Design: Define endpoints and payloads for reserving, committing, and adjusting token consumption.
2. Architecture & Data Storage: Detail the data store choice (e.g., Redis clusters, local memory, Cassandra) and the rate-limiting algorithm adapted for token units (e.g., Token Bucket, Sliding Window Counter).
3. Reserve/Commit Lifecycle: Explain how you prevent token leakage if an agent crashes midway through generation.
4. Distributed Scaling & Consistency: Explain how quotas are shared across distributed limiter nodes and multiple geographic regions.
5. Trade-offs: Evaluate strict transactional consistency vs. eventual consistency with token overage allowances.