Katkat
Back

Design a Grounded LLM Assistant for Case Preparation

DistylSoftware Engineer · Technical Screen
HardSoftware EngineerTechnical ScreenML System Design

Last updated

Design an enterprise-grade, retrieval-grounded LLM assistant for litigation teams to summarize discovery documents and draft case briefs with strict matter isolation, citation provenance, contradiction detection, and injection guardrails.

Scenario

Litigation teams handling complex disputes must analyze hundreds of thousands of discovery files—pleadings, contracts, emails, and deposition transcripts. Design an enterprise ML system that ingests these matter files, constructs factual chronologies, surfaces contradictory testimony, and drafts case-preparation briefs for attorney review. The system must act strictly as an analytical drafting aid, never issuing ungrounded conclusions or executing external actions.

Requirements

Functional Requirements

  • Matter-Isolated Document Ingestion: Ingest multi-format documents (scanned PDFs, OCR text, email threads, deposition transcripts) mapped strictly to distinct legal matters.
  • Passage Provenance & Precise Citations: Every generated factual statement must cite exact source documents down to page, paragraph, or deposition line/Bates number.
  • Contradiction Detection: Explicitly detect and surface conflicting statements across witnesses or documents regarding the same event or claim.
  • Strict Guardrails: Disallow legal advice or unauthorized external actions; enforce a read-only, human-in-the-loop review workflow.

Non-Functional & Security Requirements

  • Indirect Prompt Injection Defense: Discovery documents may contain adversarial text designed to hijack model instructions. The system must neutralize these payloads.
  • Auditability & Retention: Support immutable query/response audit logging, matter-level encryption keys, and hard retention/deletion policies (e.g., litigation holds).
  • High Precision & Low Hallucination: Citation precision must exceed 99%; unverified or hallucinated facts are unacceptable.

Scale & Performance Assumptions

  • Typical matter: 50,000 documents (~250,000 pages). Large matters: Up to 1,000,000 documents.
  • Platform load: 50 concurrent matters, peak query volume of 100 QPS.
  • Ingestion throughput: Process 10,000 pages in under 30 minutes.
  • Interactive Q&A latency: Under 4 seconds to first token; streaming responses for long-form draft generation.

What Your Design Should Cover

  1. End-to-End Architecture: Document processing, chunking with structural metadata (Bates numbers, speaker tags), hybrid search (dense + lexical), and synthesis pipeline.
  2. Grounding & Provenance Engine: How the model is constrained to generate verifiable claims and how citations are mapped back to source bounding boxes or text spans.
  3. Contradiction Analysis: Strategy to identify factual divergence across disparate documents (e.g., witness timeline discrepancies).
  4. Security & Injection Mitigation: Structural separation of instructions vs. untrusted document context, multi-tenant matter isolation, and prompt injection filtering.
  5. Evaluation Framework: Offline metrics and automated evaluation pipelines for retrieval recall, faithfulness, citation precision, and toxicity/advice refusal.

Hints

Hint 1

Retain hierarchical document metadata (matter ID, Bates number, deponent, page, line) directly inside chunk payloads to ensure citations are deterministic rather than model-guessed.

Hint 2

Mitigate indirect prompt injection by isolating untrusted document text in structured envelopes (such as XML tags or JSON schemas) and using a secondary, fast verification model to inspect retrieved chunks before prompt assembly.

Hint 3

Contradiction detection requires comparing discrete factual claims: consider extracting structured event tuples (subject, action, time, assertion) before running contradiction classification.

Follow-up questions

  • How would you architect the ingestion pipeline to handle deduplication of email threads without dropping critical contextual replies or differing recipient lists?
  • How do you evaluate hallucination rates systematically when real-world matter ground truth is expensive and protected under attorney-client privilege?
  • If an opposing party's discovery production includes white-text prompt injection intended to force the assistant to concede liability, how does your system detect and quarantine it?