Katkat
Back

Design a Conversational AI Assistant

OpenAISoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSystem Design

Last updated

This question tests the ability to design a low-latency, scalable architecture for an LLM-based conversational assistant capable of handling concurrent user requests and streaming responses.

Design a Conversational AI Assistant

Design a backend architecture for a conversational AI assistant similar to ChatGPT. The system needs to support millions of concurrent users, handle real-time streaming of text tokens, and maintain conversation state.

Requirements

  • Functional: Users must be able to start new chats, send prompts, and receive streaming token responses. History should persist across sessions.
  • Non-Functional:
    • Low latency for the first token generation (time-to-first-token < 500ms).
    • High availability and scalability for fluctuating traffic.
    • Maintainability of conversation context.
  • Scale: Assume 100k requests per second (RPS) at peak and a database of billions of past conversations.

Expectations

Your design should cover:

  1. High-level Architecture: How components like the load balancer, API gateway, model inference service, and database interact.
  2. Inference Serving: How to manage the resource-heavy LLM inference process, including handling batching or model sharding.
  3. State Management: Where to store conversation history and how to fetch it efficiently without introducing latency.
  4. Scalability Strategies: How to handle spikes in traffic and ensure the system remains responsive.

Provide a high-level diagram description and explain your choice of data storage (SQL vs. NoSQL) and caching strategies.

Hints

Hint 1

Consider using an asynchronous task queue or a streaming protocol like WebSockets or Server-Sent Events (SSE) to handle long-running model inference.

Hint 2

Think about how you would implement a distributed caching layer (e.g., Redis) to store recent conversation history and minimize database read pressure.

Hint 3

Address the 'context window' problem: since LLMs have token limits, how will you truncate or summarize conversation history before sending it to the model?

Follow-up questions

  • How would you handle model versioning and A/B testing for different LLM checkpoints without downtime?
  • If inference costs are prohibitive, what strategies (e.g., request caching, semantic caching) would you implement to reduce redundant computation?
  • How would you monitor the system to detect if the model begins 'hallucinating' or producing toxic content in real-time?