Design an Agent Harness and Evaluation System

Design a harness and evaluation platform for a tool-using AI agent. The agent receives a user task, may take multiple model and tool steps, and returns a final result. The platform must support repeatable offline evaluation, safe execution, failure diagnosis, and controlled comparison of agent versions.

Focus on the harness and evaluation design rather than model training or low-level model serving. State a representative task domain and make assumptions explicit.

Constraints & Assumptions

• Tools may be stateful, slow, unavailable, or capable of destructive actions.
• Some outcomes can be checked deterministically; others need rubric-based or human review.
• Model outputs are stochastic and costs vary by trajectory length.
• Evaluation data may contain sensitive information and must not leak into prompts or logs unnecessarily.
• A candidate version must not mutate production state during offline evaluation.

Clarifying Questions to Ask

• What agent tasks and failure costs matter most?
• Which tools and permissions exist, and can they be simulated or replayed?
• What is the release decision the evaluation must support?
• Which dimensions matter besides task success, such as latency, cost, safety, and user effort?

Hints

• Separate task specification, execution, observation, grading, and reporting.
• Preserve enough trajectory data to reproduce failures without storing secrets indiscriminately.
• Treat evaluator reliability as something that must itself be measured.