Katkat
Back

Design an LLM Agent System That Revises a Contract to Comply with Dozens of Rules

HarveySoftware Engineer · Onsite
MediumSoftware EngineerOnsiteML System Design

Last updated

Design an LLM agent system that revises a given contract so it complies with several dozen given legal rules. It tests decomposing work across model calls, representing edits so they can be traced and merged, resolving overlapping or conflicting changes, per-rule verification, and cost and latency trade-offs.

In an onsite system design round, you are given two inputs: a contract, and a list of several dozen legal rules. Each rule states a requirement that the contract must comply with. Design an LLM-based agent system that revises the contract so that it complies with every rule, and returns the revised contract.

Walk through how the work is structured and coordinated, how the system handles rules that affect the same parts of the contract, how it checks that the result is correct, and what it costs in latency and model usage.

Constraints and Clarifications

  • The rules and the contract are both provided; the system does not have to find or interpret external law.
  • There are several dozen rules, and each one must be satisfied by the returned contract.

Clarifying Questions

  • How long is the contract compared with the model's context window?
  • Are the rules independent, or can two rules conflict? If they conflict, is there a priority order?
  • Should the output be a clean revised contract, tracked changes with a reason for each edit, or both?
  • Will a lawyer review the output before it is used, and what latency and cost per contract are acceptable?
  • Must text that no rule requires changing be left exactly as it was?

Hints

Hint 1

One call, dozens of obligations: Think about how reliably a single model call can satisfy dozens of independent requirements at once, and what smaller unit of work could be carried out and checked on its own.

Hint 2

Where the work collides: Decide how the system would notice that two separate changes touch the same text, and what decides the final wording there.

Follow-up questions

  • Two rules genuinely contradict each other for one clause. What should the system do?
  • A rule changes a defined term that is used throughout the contract. How does your design handle an edit whose effects are global?
  • How would you evaluate the system before lawyers rely on it?
  • The same rule set is applied to thousands of contracts. What can be cached or precomputed?