Katkat
Back

Search Large Vehicle Records with Bounded Memory

NuroSoftware Engineer · Onsite
HardSoftware EngineerOnsiteSystem Design

Last updated

Design a petabyte-scale search engine for autonomous vehicle records supporting multidimensional range queries, bounded-memory query execution, and stable streaming pagination.

Scenario

Autonomous delivery vehicles generate immense volumes of perception logs, telemetry frames, and operational metrics. Engineering and safety teams need a centralized search platform to query vehicle records across time, geographic regions, and vehicle telemetry metrics (e.g., "find events where disengagement = true, speed > 25 mph, and location is within a 5 km bounding box in Mountain View during October").

The system must store petabytes of data, enforce a strict one-year retention policy, and execute range searches reliably without nodes running out of memory (OOM).

Requirements

Functional Requirements

  • Multidimensional Range Query: Query records by time interval, spatial bounding box (or H3/S2 cell), and scalar attributes (e.g., vehicle ID, speed, event triggers).
  • Stable Result Streaming & Pagination: Return search results deterministically across distributed workers without maintaining unbounded state per client query.
  • Data Ingestion & Retention: Ingest continuous streams of vehicle records, index them selectively, and automatically evict records older than 365 days.

Non-Functional Requirements & Scale

  • Scale: Fleet of 1,000+ vehicles generating a combined 15 TB/day of structured log records (~5.5 PB stored over a 1-year window).
  • Query SLA: p95 latency under 2 seconds for selective queries (= 50 MB/s for large analytical scans.
  • Bounded-Memory Execution: Search nodes must operate within fixed memory ceilings (e.g., 4 GB heap per query worker) regardless of matching result set size.
  • Consistency: Eventual consistency with an ingestion-to-query visibility lag under 60 seconds.

What Your Design Should Cover

  1. Data Model & Storage Layout: Storage format (columnar vs. row), partitioning strategy (time-space tiling), and file format optimizations.
  2. Indexing Strategy: Selective indexing, sparse zonemaps (min/max pruning), and spatial indexing tradeoffs for petabyte scale without index bloat.
  3. Query Engine Architecture: Coordinator-worker topology, distributed scatter-gather, bounded-memory streaming execution, and stable cursor generation.
  4. Lifecycle & Retention: Partition drops, compactions, and handling delayed or out-of-order log uploads from vehicles.

Hints

Hint 1

Partition data hierarchically by date and coarse geospatial cells (e.g., S2 level 10 or H3 resolution 7) to prune the vast majority of files before scanning indexes.

Hint 2

Rely on block-level zonemaps (min/max statistics) and dictionary encoding inside columnar formats (like Parquet or custom formats) rather than maintaining massive global B-trees or inverted indexes.

Hint 3

Implement stateful or token-based cursors that encode the last-seen sort key and file offset to stream results without loading full result sets into coordinator memory.

Follow-up questions

  • How does your query engine handle deep pagination (e.g., requesting page 1,000) without degrading into full scans or blowing worker heap?
  • If vehicles intermittently lose cellular connection and upload 48 hours of buffered data at once, how does out-of-order ingestion impact file compaction and query consistency?
  • How would you tier data between high-performance local NVMe storage and object storage (e.g., S3/GCS) to minimize cost while keeping query latency low for recent data?