Design and Evaluate a Retrieval-Augmented Generation Pipeline
HarveySoftware Engineer · Onsite
Last updated
Design a RAG pipeline with document chunking, retrieval recall evaluation, grounded answers, permission-aware context, and versioned index updates.
Design a retrieval-augmented generation pipeline over a document collection. Explain document chunking, retrieval, context assembly, and how you would evaluate retrieval recall separately from answer quality.
Constraints & Assumptions
- The system answers questions using a versioned document collection and should identify the evidence supporting its answers.
- The source calls out chunking and evaluation/recall metrics but does not provide a document domain, model, scale, or quality target. State any baseline choices explicitly.
- Treat documents as untrusted data, and preserve any access restrictions when selecting context.
- Explain how the system responds when sufficient evidence is absent; a fluent answer alone is not a success criterion.
Clarifying Questions to Ask
- What kinds of documents and questions must be supported, and how often does the corpus change?
- Is relevance defined at the document, passage, or supporting-evidence level?
- Which users may retrieve which documents, and can those permissions change?
- What latency and context-size limits constrain the amount of retrieved material?
Hints
Hint 1
Measure the pipeline in stages: A correct answer can fail because the evidence was never indexed, was not retrieved, was removed during context assembly, or was ignored during generation. Design evaluation that separates those cases.
Follow-up questions
- Why might smaller chunks improve one retrieval metric but harm the final answer?
- How would stale or deleted documents be removed from both retrieval and cached responses?
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.