Katkat
Back

Design an AI Gateway

DecagonSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSystem Design

Last updated

Design a high-throughput, multi-tenant AI Gateway that provides a unified API, intelligent routing, token-aware rate limiting, and observability across multiple upstream LLM providers.

Scenario

Enterprise AI platforms rely on a mixture of public foundation models (e.g., OpenAI, Anthropic, Bedrock) and proprietary, self-hosted open-source models (e.g., vLLM, Ollama). Relying on client applications to directly interact with heterogeneous provider APIs creates tight coupling, complicates cost control, and increases vulnerability to provider outages.

Design an enterprise-grade AI Gateway that acts as an intelligent reverse proxy layer between client applications and downstream model providers.

Requirements

Functional Requirements

  • Unified Interface: Provide a single OpenAI-compatible API surface for chat completions, embeddings, and real-time streaming via Server-Sent Events (SSE).
  • Dynamic Model Routing & Fallbacks: Support rule-based and latency/cost-based routing across providers with automatic fallback (e.g., route to Claude 3.5 Sonnet if GPT-4o returns 429 or 5xx).
  • Tenant Authentication & Rate Limiting: Enforce API key authentication, request-per-minute (RPM), and token-per-minute (TPM) quotas per tenant.
  • Cost Tracking & Telemetry: Attribute exact token usage (input, output, cached) and dollar cost to individual tenants and teams in near real time.
  • PII Redaction & Sanitization: Inspect and scrub sensitive data from prompts before forwarding to external providers.

Non-Functional Requirements

  • Low Overhead: Gateway proxy latency overhead must be under 15ms (P99), excluding downstream model generation time.
  • High Availability: 99.99% availability.
  • Scalability: Must support up to 10,000 requests per second (RPS) peak, with over 70% of requests streaming responses.

What to Cover

  1. Architecture & Request Lifecycle: Diagram the gateway components (ingress, auth, routing pipeline, streaming handler, telemetry dispatcher).
  2. Streaming & Backpressure: Explain how the gateway processes SSE streams while tracking token counts and latency metrics without buffering the entire payload.
  3. Quota & Rate Limiting Strategy: Detail the algorithm and data store used to enforce both RPM and TPM limits in a distributed environment.
  4. Resiliency & Fallbacks: Describe circuit breaker patterns, retry strategies, and mid-stream failure handling.
  5. Data Model: Outline the schema for tenant configurations, routing rules, and audit/usage records.

Hints

Hint 1

Differentiate between request-rate limits (RPM) and token-rate limits (TPM), accounting for the fact that token counts are not fully known until after generation completes.

Hint 2

Consider how Server-Sent Events (SSE) streaming complicates standard reverse proxy middleware like response buffering, token counting, and mid-stream fallback handling.

Hint 3

Use a circuit-breaker pattern alongside health-check probes so the gateway quickly stops routing to an upstream provider experiencing degraded latency or elevated 5xx errors.

Follow-up questions

  • How would you handle fallback routing if the primary LLM provider fails mid-stream after having already yielded 50 tokens to the client?
  • How can you implement semantic caching in the gateway to bypass LLM calls entirely for duplicate or semantically identical queries?
  • How would you design the rate limiter to enforce strict Token-Per-Minute (TPM) limits across thousands of concurrent gateway nodes without Redis becoming a centralized bottleneck?