Katkat
Back

Design an Autonomous-Vehicle Perception Pipeline

NVIDIASoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSystem Design

Last updated

Tests system design principles for real-time, safety-critical autonomous vehicle perception, focusing on multi-sensor hardware synchronization, zero-copy memory pipelines, bounded-latency neural inference, and fail-operational degradation.

Scenario

You are tasked with designing the real-time onboard camera perception pipeline for an autonomous vehicle (AV) powered by an embedded compute platform (such as NVIDIA DRIVE Orin / Thor). The pipeline is responsible for ingesting camera data and outputting tracked 3D obstacles, free space boundaries, and lane geometry to downstream motion planning modules.

Requirements

Functional Requirements

  • Synchronized Capture: Capture frames from multiple surround-view cameras with sub-millisecond temporal alignment.
  • Preprocessing & Calibration: Perform demosaicing, auto-exposure/white balance adjustments, undistortion, and rectification using extrinsic/intrinsic calibration data.
  • Neural Inference: Execute multi-task perception models (3D object detection, semantic segmentation, lane line extraction).
  • Fusion & Tracking: Fuse detections across overlapping camera views into a unified Bird's-Eye-View (BEV) or 3D ego-coordinate system and track objects across consecutive frames.
  • Safe Degradation: Detect sensor degradation (e.g., lens occlusion, sensor failure, thermal compute throttling) and safely transition the vehicle into a degraded or fallback operational mode.

Non-Functional Requirements

  • Bounded Latency: Hard real-time deadline of at most 40 ms from physical photon exposure to output delivery.
  • Throughput: Process all camera streams continuously at 30 FPS without dropping frames under nominal operating conditions.
  • Deterministic Memory & Compute: Zero dynamic memory allocations during runtime; zero-copy data passing between hardware blocks (Camera Serial Interface -> ISP -> GPU -> Downstream IPC).
  • Safety & Reliability: Adhere to ISO 26262 principles (ASIL-B/D considerations) for fault isolation and watchdogs.

Scale & Constraints

  • Sensor Rig: 8x 8-megapixel (3840x2160) cameras streaming at 30 FPS via GMSL2.
  • Bandwidth: Aggregate raw uncompressed data rate of ~4.5 to 6.0 GB/s.
  • Compute Budget: Dedicated embedded system with hardware accelerators (ISP, Deep Learning Accelerators (DLA), and GPU cores) sharing unified memory (e.g., 64 GB LPDDR5).

Key Topics to Cover

  1. Hardware Ingestion & Time Sync: Trigger mechanisms (PTP/IEEE 802.1AS, hardware PWM trigger lines), GMSL deserializers, and DMA into ring buffers.
  2. Zero-Copy Data Flow: Unified memory architecture, RDMA/CUDA IPC, and pipelining across CPU, ISP, and GPU.
  3. Inference & Concurrency: Concurrency model (CUDA streams, DLA partitioning), batching vs. streaming inference, and trade-offs between BEV feature-level fusion vs. late bounding-box fusion.
  4. Health Monitoring & Fault Handling: Detecting stale frames, lens blockage, sensor desync, and executing bounded-time fallback strategies.

Hints

Hint 1

Consider using hardware trigger lines (e.g., GPIO/PWM pulses from a micro-controller or master clock) combined with PTP timestamps rather than software polling to guarantee frame alignment.

Hint 2

Eliminate memory copy overhead by utilizing pinned unified memory buffers (e.g., NvMedia / CUDA inter-process communication) to stream frames directly from the deserializer to GPU memory.

Hint 3

Evaluate feature-level BEV fusion versus late object-level fusion, analyzing how each approach handles camera misalignment or missing sensor feeds.

Follow-up questions

  • How does the pipeline handle a single camera dropping frames or lagging without stalling the entire multi-camera inference loop?
  • How would you detect and handle camera degradation such as lens condensation, mud splatter, or sudden glare from oncoming high beams?
  • If the onboard compute unit enters a thermal throttling state, how should the pipeline dynamically degrade quality to preserve latency constraints?