Design a Highly Available Conversational AI Service

Design a conversational AI service that accepts a user message and streams a generated response. Begin with a deliberately minimal version that does not persist conversation history. Then evolve the design to support higher scale, stronger availability, fault tolerance, and useful operational metrics. State which later features require persistence rather than assuming them from the start.

Constraints & Assumptions

• Model inference is a separate dependency with variable latency and possible overload or failure.
• Responses are streamed and clients may disconnect or retry.
• The first version may treat each request independently.
• Safety and authorization checks remain required even in the minimal version.
• Capacity is finite; the system needs explicit overload behavior.

Clarifying Questions to Ask

• Is the first release text-only, and what request/response size limits apply?
• What latency, availability, and geographic requirements matter?
• Which safety, tenant isolation, and data-retention policies apply?
• When conversation history is later added, is cross-device continuity required?

Part 1: Minimal Stateless Path

Describe the request flow, streaming protocol, validation, safety checks, inference call, cancellation, and error behavior without storing conversation history.

What This Part Should Cover

• Authentication, quotas, input limits, request IDs, and policy enforcement.
• A gateway/orchestrator that invokes inference and relays bounded streaming output.
• Cancellation propagation and clear treatment of retries after partial streams.

Part 2: Scale, Availability, and Fault Tolerance

Explain admission control, load balancing, dependency failure handling, regional design, and what changes when persistent conversations are introduced.

What This Part Should Cover

• Capacity-aware routing, queues with bounds, backpressure, and load shedding.
• Timeouts, circuit breaking, retry limits, and no unsafe retry after unknown partial work.
• Durable message ordering and idempotency only when history becomes a requirement.
• Failure-domain isolation and graceful degradation.

Part 3: Metrics and Verification

Define service-level signals and tests that distinguish frontend streaming, orchestration, and model-provider problems.

What This Part Should Cover

• Time to first token, time to last token, generation rate, errors, cancellations, and queue delay.
• Capacity, saturation, safety outcomes, dependency health, and cost indicators.
• Load, fault-injection, disconnect, retry, and regional-failure tests.