Design an LLM-Based Conversational Assistant (Chatbot)

Scenario

Design a consumer-facing conversational AI assistant (similar to Meta AI) integrated across messaging platforms. The assistant must handle open-ended dialogues, provide factual and up-to-date answers via web search, invoke external tools (e.g., calendar, weather, reminders), and maintain strict safety and policy adherence.

Requirements

Functional Requirements

• Multi-turn dialogue: Maintain coherent context, track user intent, and resolve coreferences across turns.
• Factual grounding (RAG): Retrieve real-time information from external web indices or internal knowledge bases when questions require up-to-date or domain-specific facts.
• Tool use & API calling: Parse intent to execute external APIs structured as function calls and integrate results into natural language responses.
• Safety & Moderation: Detect and block toxic, harmful, or policy-violating queries and generations while minimizing false positive refusals.

Non-Functional Requirements

• Latency: Time-to-First-Token (TTFT) < 600 ms; Inter-Token Latency (ITL) < 35 ms per token (streaming response).
• Scale: 150 million daily active users (DAU), averaging 10 turns per user daily. Peak query traffic of ~40,000 requests per second (RPS).
• High Availability: 99.99% uptime globally across multiple data center regions.

Scale & Resource Assumptions

• Average input prompt: 800 tokens (including system prompt, retrieved context, and multi-turn history).
• Average generated response: 250 tokens.
• Target primary generation model: ~70B parameter dense model or a Mixture-of-Experts (MoE) equivalent.

Key Architectural Points to Cover

1. Model Pipeline & Post-Training: Architecture choices (Dense vs. MoE), Supervised Fine-Tuning (SFT) data collection, Preference Optimization (RLHF / DPO / PPO), and Constitutional AI / Red-teaming loops.
2. RAG & Tool Orchestration: Query rewriting, retrieval router (deciding when to search vs. answer directly), vector indexing vs. hybrid search, and prompt synthesis.
3. Serving & Inference Optimization: Decoupling prefill and decode stages, KV-cache management (e.g., PagedAttention), continuous batching, quantization (e.g., FP8/INT4), and speculative decoding.
4. Safety & Guardrails: Pre-inference moderation (input classifier), on-the-fly token checks, and post-generation safety verification.
5. Evaluation & Monitoring: Online metrics (session length, user thumbs-up/down, tool success rate) and offline benchmarks (factuality, instruction following, safety refusal rates).