Katkat
Back

Design a Large-Scale Streaming AI Product Feature

OpenAISoftware Engineer · Onsite
HardSoftware EngineerOnsiteSystem Design

Last updated

Design an end-to-end streaming architecture for an AI product feature serving millions of concurrent users. The focus is on low-latency token streaming, connection lifecycle management, data persistence, and graceful failure recovery.

Product Scenario

You are tasked with designing the core backend architecture for a high-traffic AI conversational workspace (similar to an interactive assistant or coding canvas). Users submit prompts, and the system streams generated tokens back in real time. Users can interrupt/cancel responses mid-generation, edit prompts to fork conversations, and reconnect seamlessly if network connectivity drops.

Requirements

Functional Requirements

  • Token Streaming: Stream text tokens to clients as they are generated by upstream inference clusters.
  • Stream Control: Support immediate cancellation of in-flight generations to prevent wasted GPU compute.
  • Resilience & Resumption: Allow clients that drop connection to reconnect and resume streaming from their last received token offset.
  • Conversation History: Persist complete user turns and model outputs accurately, even if generations are canceled or network interrupts occur mid-stream.

Non-Functional Requirements & Scale

  • Scale: 10 million Daily Active Users (DAU), 150,000 peak concurrent active streams.
  • Latency: Time-to-First-Token (TTFT) p95 < 800ms; inter-token latency p95 < 50ms.
  • Reliability: 99.99% uptime for conversation retrieval; graceful degradation during inference backpressure.
  • Cost Efficiency: Canceled streams must propagate upstream to kill model inference within 200ms.

What to Cover

  1. API & Protocol Choice: Evaluate Server-Sent Events (SSE), WebSockets, and HTTP/2 chunked transfer. Define the client-to-gateway and gateway-to-model API contracts.
  2. System Architecture: Component diagram from edge gateways to inference orchestrators, token buffers, and databases.
  3. Data Model & State: Schema for conversations, messages, and ephemeral streaming session states.
  4. Failure Recovery & Edge Cases: How the system handles client disconnects, inference worker crashes, and deduplication of retried requests.
  5. Observability: Key metrics to monitor latency pipelines (TTFT, token generation speed, connection drop rates).

Hints

Hint 1

Evaluate whether your client interaction requires full-duplex communication (WebSocket) or primarily server-to-client pushing with standard HTTP POST requests (SSE or HTTP/2).

Hint 2

To avoid wasting expensive GPU capacity, design an explicit cancellation heartbeat and upstream abort signal propagation (such as gRPC context cancellation) between edge servers and inference nodes.

Hint 3

For connection drop recovery, decouple token generation from the client transport connection using an intermediate cache or message broker indexed by monotonic token sequence numbers.

Follow-up questions

  • How would you implement fair-share rate limiting when users can hold open persistent streaming connections for long periods?
  • If an inference worker node crashes mid-stream, how do you handle seamless failover without showing the user an error or regenerating from scratch?
  • How does your architecture handle multi-region deployments when the model inference clusters are concentrated in regions with available GPU capacity?