Katkat
Back

Design a RAG question-answering system

HarveySoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

Design an enterprise-grade Retrieval-Augmented Generation (RAG) system capable of answering complex natural language queries over millions of internal documents with high precision, low latency, and verifiable citations.

Scenario

You are asked to design an enterprise-grade Retrieval-Augmented Generation (RAG) question-answering system for a knowledge-intensive organization (such as a legal or financial firm). The platform must ingest diverse client documents, allow users to ask complex questions, and provide accurate, synthesized answers where every factual assertion is linked to an exact source citation.

Requirements

Functional Requirements

  • Document Ingestion: Ingest unstructured and semi-structured documents (PDF, DOCX, scanned text) with automated parsing, metadata extraction, and chunking.
  • Semantic & Keyword Retrieval: Retrieve the most relevant passages matching user intent, domain-specific jargon, and exact keywords.
  • Answer Generation & Citations: Generate concise, hallucination-resistant answers with document-level and chunk-level citations (e.g., document name, page, and paragraph).
  • Access Control: Enforce tenant isolation and document-level Role-Based Access Control (RBAC) so users only see information they are authorized to view.

Non-Functional Requirements

  • Latency: P95 response latency < 2.5 seconds for the first token (using streaming responses).
  • Scale: 10 million documents (averaging 5 pages each, ~50 million chunks total), with 10,000 document updates/additions per day.
  • Throughput: Support 100 peak query QPS and up to 50 concurrent ingestion workers.
  • Freshness: Ingested or modified documents must be searchable within 60 seconds.
  • Reliability & Consistency: Zero data leakage across tenants; high attribution fidelity (no unsupported claims).

What Your Design Should Cover

  1. Data Ingestion & Indexing Pipeline: Parsing, OCR handling, semantic chunking strategies, embedding generation, and metadata association.
  2. Storage Architecture: Selection and schema design for vector databases, traditional inverted indexes (BM25), document stores, and relational metadata stores.
  3. Retrieval & Re-ranking: Hybrid search (sparse + dense), reciprocal rank fusion (RRF), cross-encoder re-ranking, and tenant/ACL filtering (pre-filtering vs. post-filtering).
  4. LLM Orchestration & Generation: Prompt design, context window budgeting, citation verification, guardrails, and streaming response delivery.
  5. Evaluation & Monitoring: Metrics to assess retrieval recall vs. generation quality, drift detection, and latency bottlenecks.

Hints

Hint 1

Consider combining sparse keyword search (BM25) with dense vector embeddings to handle both domain-specific legal/technical jargon and broader semantic meaning.

Hint 2

Evaluate pre-filtering versus post-filtering for document-level ACLs in the vector store, considering the trade-offs in search latency and recall accuracy.

Hint 3

Address how chunk boundaries affect context; hierarchical chunking or parent-document retrieval can help feed sufficient surrounding context to the LLM without diluting embedding quality.

Follow-up questions

  • How would you implement an automated verification step to flag or discard generated statements that lack explicit support in the retrieved context?
  • If an organization dynamically revokes a user's access to 100,000 documents, how do you ensure the vector search immediately respects the change without full re-indexing?
  • How would you decouple and independently evaluate retrieval performance (e.g., Recall@K, MRR) versus generation fidelity (e.g., faithfulness, answer relevance)?