Katkat
Back

Design a Rule-Guided Document Evaluation Agent

HarveyAI Agent · Onsite
MediumAI AgentOnsiteML System Design

Last updated

Design an enterprise-grade document evaluation agent that evaluates legal and policy compliance against structured playbooks, producing traceable citations, revision proposals, and tier ratings.

Scenario

Enterprise legal teams need an automated system to evaluate uploaded contracts and policy documents against internal playbooks. The system must categorize each document into a compliance tier (e.g., Approved, Acceptable with Deviations, High Risk/Escalate), generate exact redline/revision suggestions, trace every finding directly to specific contract text spans and rule definitions, reconcile disagreements across domain-specific specialist agents, and provide human-in-the-loop override capabilities.

Requirements

Functional Requirements

  • Document Ingestion & Chunking: Parse complex documents (PDF, DOCX) while maintaining absolute character and visual span coordinates.
  • Rule Evaluation Engine: Combine deterministic evaluations (e.g., regex patterns, specific clause presence, date boundaries, numeric thresholds) with semantic model judgments (e.g., indemnity risk, non-compete reasonableness).
  • Span-Level Traceability: Every finding, score, and revision suggestion must link to an immutable rule ID/version and an exact source span in the document.
  • Multi-Agent Specialist Coordination: Support parallel evaluations by specialist agents (e.g., IP specialist, Liability specialist, Data Privacy specialist) and resolve conflicting findings.
  • Human Override & Feedback: Enable human reviewers to accept, reject, or modify agent findings and overall tier ratings, storing corrections for auditing and model evaluation.

Non-Functional Requirements & Scale

  • Scale: 10,000 contract evaluations per day, with peak traffic of 1,200 documents per hour.
  • Document Size: Average 25 pages (~12,000 words / ~16,000 tokens), up to 150 pages.
  • Latency: P95 response time under 30 seconds for standard documents (asynchronous polling or webhook delivery).
  • Precision & Attribution: Groundedness/attribution rate of 99.5%+; hallucinated citations or broken span offsets are strictly intolerable.

Key Discussion Points

  1. API & Data Models: Define schemas for Rule, DocumentSpan, Finding, and EvaluationReport.
  2. System Architecture: Detail the pipeline from document parsing and AST/offset generation, deterministic pre-filtering, LLM specialist execution, through conflict arbitration.
  3. Rule Versioning & Auditability: How to guarantee reproducible evaluations even if rules change over time.
  4. Conflict Resolution: Mechanism to synthesize divergent risk assessments between specialist evaluators (e.g., rule-based priority matrix vs. meta-adjudicator model).
  5. Evaluation & Quality Assurance: Offline benchmark methodology for measuring precision of text spans, rule adherence, and suggestion quality.

Hints

Hint 1

Separate the execution into deterministic rule filters (code-based) and probabilistic LLM evaluators to minimize latency and inference cost.

Hint 2

Preserve document layout and exact character offset indices during the initial parsing step so model outputs can be mapped back to character-precise coordinates without re-searching the text.

Hint 3

Model the conflict reconciliation step using an explicit priority hierarchy (e.g., highest severity wins or deterministic veto) before falling back to a general meta-evaluator LLM.

Follow-up questions

  • How do you handle cases where an LLM hallucinates an offset or generates a quote that does not exactly match the parsed document source?
  • If a customer updates their compliance playbook mid-day, how do you handle running jobs and historical audit trails for existing contracts?
  • How would you evaluate and continuously test the reliability of specialist agents without relying entirely on manual human review?