Katkat
Back

Design a Production Retrieval-Augmented Generation Pipeline

NetappSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteML System Design

Last updated

This problem tests the ability to design an end-to-end RAG system, focusing on document processing, embedding strategies, retrieval mechanisms, and ensuring accuracy through grounding.

Problem Statement

Design a production-grade Retrieval-Augmented Generation (RAG) pipeline for a corporate knowledge base consisting of millions of PDF and Markdown documents stored in cloud object storage (e.g., S3). The system must allow users to query these documents and receive concise, grounded answers from a Large Language Model (LLM).

Functional Requirements

  1. Document Ingestion: Automatically process updates to the document store.
  2. Retrieval: Efficiently retrieve relevant context based on a natural language query.
  3. Generation: Produce answers that cite sources from the retrieved context.
  4. Reliability: Handle partial failures and high latency.

Non-Functional Requirements

  1. Scale: Support a corpus of 10 million documents.
  2. Latency: Retrieval should be under 200ms; end-to-end generation under 3 seconds.
  3. Accuracy: Minimize hallucinations and ensure citations are verifiable.

Deliverables

  1. High-Level Architecture: Diagram the flow from document ingestion to the LLM response.
  2. Data Pipeline: Describe the ingestion workflow (chunking, embedding, vector DB selection).
  3. Orchestration: Explain how the retrieved context is passed to the LLM to maintain grounding.
  4. Evaluation: Define a strategy to measure the retrieval quality and generation accuracy.

Hints

Hint 1

Consider how variable document lengths impact your chunking strategy and whether you need overlapping chunks.

Hint 2

Think about the trade-offs between dense retrieval (embeddings) and sparse retrieval (BM25) for high-precision technical queries.

Hint 3

Address the 'garbage-in, garbage-out' problem by proposing a strategy for reranking retrieved results before passing them to the generation step.

Follow-up questions

  • How would you handle security and access control, ensuring users only retrieve snippets from documents they are authorized to view?
  • What specific metrics (e.g., RAGAS, faithfulness, relevancy) would you implement in an automated evaluation loop?
  • How would you optimize the system for cost given that LLM API calls are expensive at scale?