Katkat
Back

Design an Evaluation Platform for AI Agents Across Prompt and Model Changes

CrestaSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteML System Design

Last updated

An ML system design question about a platform that lets teams evaluate AI agents and measure how a change of system prompt or model affects quality, tool use and response speed. It tests evaluation design, scoring with references and LLM judges, statistical comparison of versions and scalable execution.

Design a system for evaluating AI agents. The users of the system are teams that develop AI agents, for example agents that automatically analyze contact center conversations to produce a report. These teams need a platform that lets them evaluate how well their agents are performing. For example, when a team changes an agent's system prompt, or the model the agent uses, it needs to measure how that change affects the agent's performance.

The platform should let teams compare agent configurations along several dimensions, such as how well the agent uses its tools, how quickly it responds, and how different system prompts and different models change the results.

Clarifying Questions

  • What does an agent look like to the platform: a configuration (model, system prompt, tools) that the platform runs itself, or an endpoint that each team hosts?
  • Is the focus offline evaluation on curated datasets, online evaluation on production traffic, or both?
  • Where do test inputs come from: curated conversations, samples of production traffic, synthetic data? Are there privacy restrictions on real conversations?
  • Is there ground truth, such as human-written reports or labeled fields, or must quality be judged without references?
  • Besides quality, which metrics matter: tool-use correctness, latency, cost?
  • Should an evaluation run be able to gate a prompt or model change before it ships, and how quickly must it finish?

Hints

Hint 1

Start from the unit of comparison: Pin down what one evaluation run compares, such as a baseline agent configuration against a candidate on the same inputs, before designing storage or scoring.

Hint 2

Scoring without one right answer: A report written from a conversation rarely has a single correct output. Decide which checks can be deterministic, which need a reference, which need a judge, and how you would come to trust that judge.

Follow-up questions

  • How would you validate an LLM judge, and detect that its behavior changed after the judge model was upgraded?
  • How would you evaluate multi-turn agents whose later tool calls depend on the results of earlier ones?
  • How would you keep evaluation cost under control as datasets and the number of teams grow?
  • How would production failures flow back into the evaluation datasets?