Katkat
Back

Design a Reliable AI Chatbot Architecture

AmazonSoftware Engineer · Technical Screen
EasySoftware EngineerTechnical ScreenML System Design

Last updated

Tests candidate's ability to design an enterprise-grade Retrieval-Augmented Generation (RAG) chatbot system featuring fine-grained authorization, hallucination guardrails, and low-latency response serving.

Scenario

Design an enterprise AI assistant for an organization like Amazon that answers employee questions across internal wikis, policy documents, and internal tool manuals. The assistant must provide accurate, multi-turn answers grounded strictly in authorized company documents while declining to answer when confident support is missing.

Requirements

Functional Requirements

  • Multi-turn Context: Maintain context and conversational history across multiple query turns.
  • Authorization-Aware Retrieval: Ensure users only retrieve context and receive answers derived from documents they are explicitly authorized to read (Access Control Lists / ACLs).
  • Faithfulness and Attribution: Generated answers must cite supporting documents; unsubstantiated claims or hallucinations must be suppressed.
  • Uncertainty Handling: When retrieval returns low-relevance results or documents do not contain the answer, the system should explicitly state that it cannot answer rather than guessing.

Non-Functional Requirements

  • Latency: Time-to-first-token (TTFT) < 1.5 seconds (P95); complete response delivered within 4 seconds.
  • Scale: 500,000 daily active employees generating approximately 5,000,000 queries per day (~60 average QPS, ~300 peak QPS).
  • Data Freshness: Corpus contains 10 million documents; document updates and permission changes must reflect in retrieval within 10 minutes.
  • Reliability & Availability: 99.9% uptime.

What Your Design Should Cover

  1. High-Level Architecture: Ingestion pipeline, vector index with metadata filtering, retrieval orchestration, LLM inference, and guardrail layers.
  2. Authorization-Aware Search: Strategy for evaluating document-level and group-level ACLs (pre-filtering vs. post-filtering vs. join at query time) while maintaining retrieval recall.
  3. Hallucination & Uncertainty Mitigation: Verification pipeline (e.g., cross-encoders, NLI models, or self-consistency checks) to flag unsupported claims before presenting output.
  4. API and Data Model: Core endpoints for the chat session and document ingestion, along with conversational state tracking.
  5. Latency Optimizations: Techniques to minimize perceived latency across retrieval, reranking, and generation.

Hints

Hint 1

Consider whether access control filtering should occur before or after vector similarity search to prevent unauthorized document leakage while avoiding degraded top-k recall.

Hint 2

Think about using a secondary verification step, such as a lightweight Natural Language Inference (NLI) model, to verify whether generated claims are strictly entailed by the retrieved context.

Hint 3

To maintain low perceived latency during multi-turn interactions, explore token streaming via Server-Sent Events (SSE) and caching frequent query-document pairs.

Follow-up questions

  • How would you handle instant revocation of a user's document permissions without needing to re-index or invalidate vector embeddings?
  • If the retrieved context contains contradictory information from two equally authoritative documents, how should the system arbitrate and respond?
  • How would you design an automated offline evaluation framework to monitor hallucination rates, retrieval recall, and citation precision across model updates?