Katkat
Back

Design a Memo Q&A Agent for a Large Law Firm

HarveySoftware Engineer · Onsite
MediumSoftware EngineerOnsiteML System Design

Last updated

This problem evaluates your ability to design a RAG-based Q&A system for sensitive legal documents, focusing on multi-stage retrieval, hallucination prevention, and data privacy.

Scenario

A large law firm needs an internal Q&A agent that allows associates to query thousands of historical legal memos, precedents, and case briefs. The system must provide accurate, cited answers while strictly adhering to internal data privacy policies (e.g., only authorized users access specific folders).

Requirements

  • Functional: Users submit natural language questions; the system returns an answer with citations to the source documents.
  • Non-functional: High precision is critical (hallucinations in legal contexts are catastrophic). Low latency is secondary to answer accuracy and grounding.
  • Scale: 50,000+ complex PDF documents, totaling roughly 100GB of text data.

Design Scope

  1. Data Ingestion: How do you parse and chunk legal documents that often contain tables, footnotes, and multi-column layouts?
  2. Retrieval Architecture: How do you ensure the system retrieves the correct legal context when a query uses domain-specific jargon?
  3. Grounding & Faithfulness: How do you minimize hallucinations and ensure the LLM strictly adheres to the retrieved context?
  4. Privacy & Governance: How do you handle document-level permissions during the retrieval process?

Hints

Hint 1

Consider using an embedding model fine-tuned on legal corpora or a hybrid search approach (combining dense vector search with BM25 keyword matching).

Hint 2

Think about the 'Re-ranking' phase: How can you use a cross-encoder to refine the top-K retrieved documents before passing them to the generative LLM?

Hint 3

Address the 'Citation' requirement: How do you map specific sentences in the LLM's response back to the unique IDs of the source chunks?

Follow-up questions

  • How would you measure the performance of this system using automated metrics like RAGAS or a 'LLM-as-a-judge' framework?
  • What strategies would you implement to update the knowledge base when a new, high-priority court ruling is published?
  • If the firm moves to a multi-tenant cloud environment, how do you prevent information leakage across different legal practice groups?