Katkat
Back

Design an Evaluation Platform for a Design-Editing Agent

FigmaSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteML System Design

Last updated

Design an automated, multimodal evaluation platform to benchmark and monitor a canvas-based design-editing agent across deterministic property updates and subjective creative directives.

Scenario

A collaborative design tool is rolling out an AI editing agent that modifies complex document scene graphs. User prompts range from deterministic, low-level adjustments (for example, "change all primary buttons to #0D99FF with 8px radius") to open-ended creative styling (for example, "redesign this landing page hero to look minimalist and spacious").

You are tasked with designing an evaluation platform to benchmark agent iterations before deployment and monitor agent performance continuously in production.

Requirements

Functional Requirements

  • Hybrid Evaluation Engine: Support two evaluation tracks:
    1. Programmatic Assertions: Exact scene-graph and AST diffing for structured edits (colors, layout constraints, token swaps).
    2. Multimodal Qualitative Assessment: Vision-and-language model (VLM) judges and heuristic scoring for open-ended design changes (aesthetic cohesion, spatial balance, prompt adherence).
  • Golden Dataset Management: Maintain versioned, immutable benchmark suites representing diverse design systems, edge cases, and protected client-style archetypes.
  • Production Telemetry Correlation: Ingest implicit online user feedback (undo, keep, manual tweak distance, re-prompt) and correlate it with offline benchmark metrics.
  • Evaluator Calibration: Support human-in-the-loop audit workflows to measure and correct evaluator bias, false positives, and drift in automated judges.
  • Extensibility & CI/CD: Provide developer CLI/APIs for pre-merge regression testing across product squads.

Non-Functional Requirements

  • Determinism & Reproducibility: Ensure identical evaluations can be re-run against historical agent checkpoints with pinned rendering engines and frozen judge configurations.
  • Throughput & Speed: Run a standard regression suite of 2,000 benchmark tasks within 20 minutes to serve as a deploy gate.
  • Cost Efficiency: Minimize inference costs for multimodal LLM evaluators without compromising signal fidelity.

Scale Assumptions

  • Golden benchmark repository: 10,000 diverse document templates and mock project states.
  • CI runs: 40–60 full benchmark runs per day across development teams.
  • Production volume: 2 million agent-assisted user actions per day generating interaction events.

What to Cover

  1. High-Level Architecture: Pipeline from document state snapshot, agent action execution in headless canvas, to metric extraction.
  2. Evaluation Framework: Separation of AST/scene-graph inspection vs. rendered pixel analysis (multimodal judges, perceptual hashing, layout heuristics).
  3. Data Model: Schemas for benchmark tasks, agent execution traces, evaluator outputs, and telemetry events.
  4. Judge Calibration: Strategies to evaluate the evaluators (e.g., Cohen's Kappa against expert designer labels, pairwise vs. rubric-based scoring).
  5. Trade-offs & Edge Cases: Managing non-deterministic agent outputs, visual rendering differences across platforms, and cost vs. accuracy trade-offs.

Hints

Hint 1

Differentiate early between what can be verified deterministically via scene-graph inspection (e.g., node count, color tokens, layout hierarchy) and what strictly requires pixel-level visual understanding.

Hint 2

Decouple the agent's target document state from live production documents by using synthetic or sanitized, frozen document snapshots that render in a headless environment.

Hint 3

For subjective evaluations, consider a cascade strategy where lightweight heuristics filter out obvious failures before invoking expensive multimodal LLM-as-a-judge models.

Follow-up questions

  • How would you detect and mitigate judge drift when the underlying visual foundation model used for evaluation is updated or deprecated by its vendor?
  • How can you quantify a user's 'tweak distance' when they accept an agent's edit but manually adjust three elements immediately afterward?
  • What strategies would you use to prevent synthetic benchmark datasets from overfitting to the agent's current generation patterns?