Katkat
Back

Design a RAG Ranking Pipeline

MicrosoftSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteML System Design

Last updated

This problem tests your ability to design a multi-stage retrieval and ranking architecture for a RAG system, focusing on balancing search accuracy, latency, and system scalability.

Problem: Designing a RAG Ranking Pipeline

You are asked to design the backend pipeline for an enterprise-grade document assistant. Users submit natural language queries, and the system must retrieve relevant information from a corpus of 10 million internal PDF and Word documents to generate an accurate response.

Requirements

  1. Functional: The system should retrieve the top-K relevant document chunks and pass them to an LLM to generate a grounded answer.
  2. Non-functional: The system must handle a latency of under 2 seconds for a query. The pipeline must scale to handle 500 queries per second (QPS).
  3. Access Control: The system must respect document-level permissions (i.e., users can only retrieve snippets from documents they are authorized to view).

Your Task

Design the architecture for this pipeline. Please address the following:

  • Indexing Pipeline: How do you process documents, chunk them, and store them for efficient retrieval?
  • Retrieval Strategy: How do you balance speed and relevance? Consider a two-stage approach (retrieval and re-ranking).
  • Access Control: How do you integrate permission checks without significantly impacting retrieval latency?
  • Evaluation: What metrics would you use to measure the quality of the retrieval phase versus the generation phase?

Include a high-level diagram or description of the components and their interactions.

Hints

Hint 1

Consider using an inverted index (like BM25) for keyword-based filtering alongside a vector database for semantic search.

Hint 2

Access control can be handled by filtering at the database level using document-id metadata or by employing an ACL-aware search index.

Hint 3

A two-stage pipeline using a lightweight retriever (dense/sparse) followed by a cross-encoder re-ranker is standard for managing accuracy vs. latency trade-offs.

Follow-up questions

  • How would you handle document updates or deletions in your index to ensure the retrieved information is always fresh?
  • If the LLM generates a hallucinated answer despite relevant retrieved context, what diagnostic steps or architectural changes would you implement?
  • How would you optimize the system if the document corpus grew to 1 billion documents?