Katkat
Back

Design ML system for self-driving perception

GoogleMachine Learning Engineer · Technical Screen
MediumMachine Learning EngineerTechnical ScreenML System Design

Last updated

This question assesses the ability to architect a real-time perception system for autonomous vehicles, focusing on sensor integration, model optimization for edge deployment, and safety-critical failure handling.

System Design: Autonomous Obstacle Perception

You are tasked with designing the perception module for an autonomous vehicle. The system must process data from multiple onboard sensors—LiDAR, high-resolution cameras, and radar—to identify obstacles and provide input to the motion planning module in real-time.

Functional Requirements

  • Object detection: Classify and localize objects (pedestrians, vehicles, cyclists).
  • Sensor Fusion: Combine heterogeneous sensor inputs to improve detection robustness.
  • Latency: The pipeline must produce an inference output every 50ms.

Non-Functional Requirements

  • Reliability: Safety-critical objects (e.g., pedestrians) must have a near-zero false-negative rate.
  • Edge Constraint: The system runs on onboard hardware with limited compute and power budget.

Your Task

Describe a high-level architecture for this system. In your answer, please address:

  1. Data Representation: How do you handle sensor calibration and time synchronization across different sensor types?
  2. Model Architecture: Discuss the trade-offs between a centralized fusion approach (processing raw data together) versus a late fusion approach (processing individual sensors and merging their predictions).
  3. Production Constraints: Given the 50ms latency requirement, how would you optimize your perception model?
  4. Failure Modes: How does the system handle sensor occlusion or degradation (e.g., heavy rain or lens blockage)?

Hints

Hint 1

Consider how temporal synchronization is maintained using timestamps or global positioning tags before feeding data into the perception pipeline.

Hint 2

Evaluate whether early fusion (concatenating feature maps) or late fusion (combining bounding boxes) better balances compute efficiency with information loss.

Hint 3

Think about quantization, pruning, or knowledge distillation as potential strategies to meet tight inference latency budgets on embedded hardware.

Follow-up questions

  • How would you design a data flywheel to identify and label 'corner cases' where the model performs poorly?
  • If the vehicle loses access to LiDAR data while driving, how would you design a graceful degradation strategy?
  • How would you measure the performance of your system in a simulation environment before deploying to real-world hardware?