Design Streaming Speech and Multi-Model Chat Systems

Redesign a batch speech-transcription pipeline to process audio incrementally and support multiple speakers. Then design an LLM chat service that streams responses, switches models when rate or token limits are reached, scales behind a load balancer, and stores conversation history safely.

Constraints & Assumptions

• Audio chunks can arrive late, be duplicated, or end mid-utterance.
• Partial transcripts may be revised as more context arrives.
• A model switch must not silently change the request's safety or tool permissions.
• Load balancers cannot rely on one process retaining all stream state.

Clarifying Questions to Ask

• What latency and final-transcript accuracy are required?
• Is speaker diarization based on separate channels or mixed audio?
• Which provider limits are per request, model, tenant, or time window?

Part 1 — Streaming transcription

Define chunking, ordering, buffering, partial and final transcript states, speaker handling, backpressure, and performance measurement.

What This Part Should Cover

• Sequence IDs and bounded reordering
• Revision-aware transcript segments
• Diarization, VAD, latency, and throughput diagnostics

Part 2 — Chat serving

Design request routing, streaming through a load balancer, model fallback, rate-limit handling, and conversation persistence.

What This Part Should Cover

• Externalized stream and request state
• Capability-aware routing and idempotency
• Durable message lifecycle and token accounting