Katkat
Back

Design a RAG Q&A Agent over Law Firm Legal Memos

HarveySoftware Engineer · Onsite
HardSoftware EngineerOnsiteML System Design

Last updated

Design an end-to-end Retrieval-Augmented Generation (RAG) system for querying confidential, complex legal memos. This evaluates domain-specific parsing, hybrid retrieval with fine-grained access control, and grounded generation with strict citation fidelity.

Scenario

A high-profile corporate law firm wants an internal AI assistant to help attorneys analyze, summarize, and cross-reference thousands of proprietary legal memos, case briefs, and regulatory opinions. Legal research requires zero tolerance for hallucinations, strict multi-hop factual consistency, and paragraph-level attribution.

Requirements

Functional Requirements

  • Natural Language Q&A: Attorneys can query the corpus in natural language (e.g., "What arguments did we use regarding material adverse effects under Delaware law between 2021 and 2023?").
  • Grounded Citations: Every factual assertion must be attributed to a specific memo, page, and paragraph with deep links to source documents.
  • Temporal & Metadata Filtering: Support filtering and relevance scoring based on jurisdiction, practice group, author, client matter, and date.
  • Strict Access Control: Enforce ethical walls and client-matter confidentiality (users must never retrieve or infer details from unauthorized matters).

Non-Functional Requirements

  • Low Latency: End-to-end latency under 3 seconds for initial token generation; total response stream completion under 8 seconds.
  • High Precision & Groundedness: Strict rejection/fallback when evidence is insufficient, avoiding hallucinated legal precedents or statutory claims.
  • Data Privacy & Compliance: Zero data leakage across matter boundaries; strict compliance with legal data retention and redaction rules.

Scale & Assumptions

  • Corpus Size: 10 million legal memos and briefs, averaging 20 pages per document (~200 million pages total, mostly PDF/DOCX with nested tables, footnotes, and complex formatting).
  • Growth Rate: ~50,000 new or updated documents per week.
  • Throughput: 100 QPS peak query volume across 5,000 active concurrent attorneys.

Key Areas to Cover

  1. Document Ingestion & Chunking: How to parse semi-structured legal text (handling headers, statutory citations, and footnotes) without fragmenting legal concepts.
  2. Indexing & Retrieval: Hybrid search architecture (sparse BM25 + dense semantic embeddings), cross-encoder reranking, and dynamic metadata/permission pre-filtering.
  3. Grounded Generation & Verification: Prompt orchestration, context window compression, and an automated verification layer to validate citation alignment before showing results to the user.
  4. Evaluation & Monitoring: Offline benchmark curation (using legal expert annotations) and online guardrails for factual drift and hallucination tracking.

Hints

Hint 1

Legal documents rely heavily on hierarchical structure (sections, sub-clauses, footnotes); standard fixed-size chunking risks splitting crucial qualifiers or statutory definitions across chunk boundaries.

Hint 2

Legal terminology often combines exact phrasing (statutes, case numbers, party names) with abstract principles, making a hybrid search (dense + lexical BM25) and reranking pipeline essential.

Hint 3

Confidentiality and ethical walls require integrating role-based access control (RBAC) directly into vector and keyword filtering stages rather than filtering post-retrieval.

Follow-up questions

  • How would you handle multi-hop queries that require synthesizing information across multiple memos spanning several years?
  • How can you design an automated verification model to validate whether the generated answer is strictly entailed by the retrieved chunks before presenting it to an attorney?
  • When a document is marked for redaction or deleted due to legal obligations, how do you guarantee immediate purge across caches, vector indices, and reranker logs?