Katkat
Back

Design a Retrieval-Augmented Generation (RAG) system

OpenAISoftware Engineer · Technical Screen
HardSoftware EngineerTechnical ScreenML System Design

Last updated

This question tests the ability to design an end-to-end Retrieval-Augmented Generation (RAG) system, focusing on balancing retrieval accuracy, latency, data freshness, and system reliability.

System Design: Enterprise RAG Service

Design an enterprise-grade Retrieval-Augmented Generation (RAG) system that allows employees to query a large, frequently updated corpus of internal company documentation (PDFs, Wikis, Slack logs).

Functional Requirements

  1. Users should receive accurate answers grounded in company documentation with citations.
  2. The system must support real-time updates: when a document is uploaded, it should be searchable within minutes.
  3. The system must enforce document-level access control (users should only retrieve results from documents they have permissions to view).

Non-Functional Requirements

  1. Latency: The time to first token should be under 2 seconds.
  2. Scale: Support a corpus of 10 million documents and 1,000 concurrent requests per second.
  3. Reliability: The system must handle document ingestion failures and LLM provider outages gracefully.

Your Answer Should Include:

  • High-level architecture: Diagram the flow from document ingestion to query execution.
  • Data Pipeline: Explain the chunking strategy and embedding model selection.
  • Retrieval Strategy: How do you ensure high precision and recall? Consider hybrid search (semantic + keyword).
  • Access Control: Propose an efficient way to filter search results by user permissions without latency spikes.
  • Observability: How do you measure the quality of retrieval and the faithfulness of the generated answers?

Hints

Hint 1

Consider using an inverted index for keyword search alongside a vector database for semantic search to handle specialized terminology better.

Hint 2

To handle access control at scale, pre-filter the vector search using metadata tags or utilize a post-retrieval filtering step with a cached permission service.

Hint 3

Implement an asynchronous processing pipeline for document ingestion using a message queue to ensure robustness and separation from the request/response path.

Follow-up questions

  • How would you handle document updates or deletions to avoid 'stale' information in your vector index?
  • If the LLM frequently hallucinates information despite having the correct context in the retrieved chunks, what strategies would you implement to mitigate this?
  • How would you re-rank retrieved documents to improve answer relevance before passing them to the LLM?