Katkat
Back

Design a Hybrid Evaluation Platform

WaymoSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteML System Design

Last updated

This problem tests your ability to design a multi-modal evaluation infrastructure that balances automated LLM-based grading with human-in-the-loop (HITL) verification to ensure high-fidelity ML model performance metrics.

Design a Hybrid Evaluation Platform

Waymo requires a robust evaluation platform to validate perception and planning models. You are tasked with designing a hybrid system that assesses model outputs using a combination of automated metrics, LLM-based judging, and human expert review.

Requirements

Functional Requirements:

  1. Pipeline Orchestration: Support batch processing of model inference logs against ground truth data.
  2. Multi-tier Grading: Provide a workflow where automated judges perform initial passes, and high-uncertainty or edge-case results are escalated to human evaluators.
  3. Evaluation Store: A system to version control evaluation prompts, model weights, and evaluator feedback to track performance regression over time.

Non-Functional Requirements:

  1. Scalability: Handle processing of thousands of driving scenarios per hour.
  2. Calibration: Mechanism to detect and correct bias in human evaluators (e.g., inter-rater reliability).
  3. Extensibility: The system must support adding new evaluation metrics (e.g., safety-critical violations, comfort metrics) without rewriting the core workflow.

Your Task

Provide a high-level architecture design that covers:

  • Data Model: How do you store evaluation results, ground truth, and the provenance of who (or what) assigned a score?
  • System Architecture: How do you handle the transition between automated judges and human experts? How is task queueing managed?
  • Quality Control: How do you measure and ensure the consistency of human evaluators vs. automated LLM judges?
  • API/Interface: What are the core endpoints or interfaces for a researcher to trigger an evaluation job and query results?

Hints

Hint 1

Consider using an asynchronous message queue to decouple the ingestion of simulation logs from the evaluation processing engine.

Hint 2

Think about 'Golden Sets'—a subset of data with known ground truth—to periodically test human evaluators for drift or inconsistency.

Hint 3

For the data model, prioritize a graph or relational schema that maintains strict versioning between the specific model iteration, the prompt used for the judge, and the final grade.

Follow-up questions

  • How would you handle a situation where the LLM judge and the human evaluator consistently disagree on a specific class of driving maneuvers?
  • If our evaluation budget is limited, what strategies would you implement to select the most impactful scenarios for human review?
  • How does your design address 'prompt drift' in LLM-based judges to ensure that evaluation results remain comparable across quarters?