Katkat
Back

Design a RAG-based assistant service

MicrosoftMachine Learning Engineer · Onsite
MediumMachine Learning EngineerOnsiteML System Design

Last updated

This question assesses the ability to architect a scalable Retrieval-Augmented Generation (RAG) system, focusing on data indexing, retrieval strategy, and mitigating common LLM issues like hallucinations and safety.

Problem Statement

Design a RAG-based document assistant for an enterprise client. The system should allow users to query a large repository of internal PDF and Markdown documents and receive AI-generated answers with citations.

Requirements

  • Functional: Users can upload documents; the system should retrieve relevant context to answer queries. The output must include links/citations to source documents.
  • Scale: The corpus consists of 500,000 documents, growing by 5,000 per day. Low latency (< 2 seconds) is expected for query responses.
  • Non-functional:
    • Implement access control: Users should only retrieve content from documents they are authorized to access.
    • Handle document freshness: Updates to existing documents must reflect in search results within 1 hour.
    • Ensure safety: Prevent the model from leaking PII or answering questions based on sensitive unauthorized documents.

Expectations

Provide a high-level architecture diagram or description covering:

  1. The ingestion pipeline (parsing, chunking, and embedding).
  2. The storage strategy (vector database vs. relational metadata store).
  3. The retrieval logic (hybrid search, re-ranking, and access filtering).
  4. A strategy to mitigate hallucinations and ensure citation accuracy.
  5. Observability and monitoring metrics for the RAG pipeline.

Hints

Hint 1

Consider how you will combine keyword-based search with semantic vector search to improve retrieval precision.

Hint 2

Think about how to store Access Control Lists (ACLs) so that they can be efficiently applied as a filter during vector search.

Hint 3

Consider implementing a 'Re-ranker' model after the initial retrieval to boost the relevance of the context window provided to the LLM.

Follow-up questions

  • How would you handle document versioning where a user asks about a specific version of a policy that has since been updated?
  • What strategies would you employ to measure the 'faithfulness' of the generated response compared to the retrieved source chunks?
  • How would you design the system to handle 'multi-hop' queries that require synthesizing information from three different documents?