Katkat
Back

Design an Agent-Orchestrated QA Pipeline from Requirements to Test Report

ByteDanceSenior SDE, Quality Platform & AI Test Automation · Technical Screen
MediumSenior SDE, Quality Platform & AI Test AutomationTechnical ScreenML System Design

Last updated

Design an end-to-end multi-agent QA platform that autonomously transforms feature requirements into validated test cases, synthetic data, executable test scripts, and traceable execution reports with human-in-the-loop oversight.

Product Scenario

Modern agile teams ship hundreds of pull requests and feature updates daily. Manual test authoring and maintenance create severe engineering bottlenecks. You are tasked with designing an AI-agent-orchestrated Quality Engineering platform for an enterprise software ecosystem (e.g., ByteDance-scale microservices and mobile/web apps). The platform ingests requirements (PRDs, Figma links, OpenAPI specs) and orchestrates a fleet of specialized LLM agents to produce verified test cases, test data, executable scripts, and comprehensive execution reports.

Functional Requirements

  • Requirement Parsing & Traceability: Ingest raw PRDs, diffs, and API schemas to generate structured, typed test cases with bidirectional traceability (Requirement -> Test Case -> Code -> Execution Run).
  • Data & Script Generation: Generate realistic, privacy-compliant test fixtures and executable test scripts (e.g., Playwright for E2E, PyTest for integration APIs).
  • Sandboxed Execution & Self-Healing: Execute generated scripts in an isolated sandbox, dynamically debugging syntax or selector errors (up to N retries) without masking genuine product bugs.
  • Human-in-the-Loop (HITL) Gates: Provide review checkpoints for high-risk test additions or destructive test data generation.
  • Report & Defect Synthesis: Produce structured test reports with execution evidence (logs, HAR traces, screenshots) and draft deduplicated bug tickets for genuine failures.

Non-Functional Requirements

  • Reliability & Determinism: Prevent hallucinations by enforcing strict schema validation (e.g., JSON Schema/Pydantic) between agent handoffs.
  • Security & Isolation: Ensure untrusted, agent-generated code cannot access production systems, exfiltrate data, or escape runtime environments.
  • Latency & Throughput: Pipeline turnaround time under 15 minutes for standard feature PRs.

Scale & Load Assumptions

  • 2,500 pull requests and feature specs processed per day.
  • Average of 20 test cases generated per feature spec.
  • Up to 50,000 automated test executions per day across distributed runners.

Key Components to Cover

  1. Architecture & Agent State Machine: Define specialized agents (Planner, Data Generator, Coder, Evaluator) and state orchestration (DAG/state graph).
  2. Schema & Artifact Contracts: Define typed schemas passed between agents to guarantee deterministic transitions.
  3. Execution Sandbox: Describe the runtime environment for executing untrusted agent code.
  4. Self-Healing vs. Bug Classification: Detail how the system differentiates between bad test scripts and actual software regressions.

Hints

Hint 1

Decompose the pipeline into specialized agents with strict typed schemas (e.g., Pydantic/JSON Schema) rather than relying on a single monolithic prompt.

Hint 2

Isolate script execution using ephemeral microVMs or unprivileged container sandboxes with network policies and mock endpoints to mitigate security risks.

Hint 3

Use static analysis (AST verification) and environment reflection (DOM trees, API response schemas) to guide self-healing retries before flagging a true defect.

Follow-up questions

  • How does the platform distinguish between an agent-authored script bug (flakiness/bad selector) and an actual regression in the underlying service under test?
  • How would you evaluate and benchmark the quality of generated test suites over time to prevent regression in agent capability?
  • How do you optimize LLM context window usage and operational token costs when ingesting large PRDs alongside hundreds of API endpoint definitions?