Katkat
Back

Design fast search for AV raw logs

NuroSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSystem Design

Last updated

This question tests the ability to architect a distributed storage and retrieval system capable of handling high-velocity, high-volume sensor telemetry data while supporting efficient time-range queries.

Problem Statement

You are designing a logging and search service for an autonomous vehicle (AV) fleet. Each vehicle continuously generates raw data streams (LiDAR, camera feeds, IMU, GPS) and perception outputs (object bounding boxes, lane detections) at roughly 500 MB per second while in operation. We need to store this data and enable engineers to search for specific time intervals across the fleet based on metadata (e.g., "find all events where a vehicle encountered a red light at an intersection") or sensor characteristics (e.g., "find logs where LiDAR noise exceeded a specific threshold").

Requirements

  • High throughput ingestion from thousands of vehicles.
  • Efficient retrieval of raw sensor data within a specified time range for any given vehicle.
  • Support for fast filtering of time segments based on indexed perception metadata.
  • Minimize storage costs while maintaining accessibility for R&D analysis.

Deliverables

  1. Propose a high-level architecture for data ingestion and storage.
  2. Design an indexing strategy to allow fast retrieval based on timestamps and object metadata.
  3. Discuss how you would handle data lifecycle (e.g., cold vs. hot storage) given the immense volume of raw sensor data.
  4. Explain trade-offs between storing raw data versus pre-processed feature vectors.

Hints

Hint 1

Consider decoupling the ingestion of massive raw binary blobs from the indexing of lightweight metadata.

Hint 2

Think about partitioned storage strategies, such as sharding by vehicle ID or time-based buckets.

Hint 3

Evaluate whether a traditional RDBMS or a time-series optimized database is better suited for the metadata index.

Follow-up questions

  • How would you handle a spike in data traffic if 1,000 vehicles return to the depot and upload logs simultaneously?
  • If we need to perform fleet-wide analytics (e.g., identifying patterns in disengagements), what tools would you use to process the data at scale?
  • How would you ensure data integrity and handle network interruptions during the log upload process?