Katkat
Back

Design street-view image ingestion and storage system

GoogleSoftware Engineer · Technical Screen
HardSoftware EngineerTechnical ScreenSystem Design

Last updated

This question tests the ability to design a high-throughput, fault-tolerant ingestion pipeline for massive geospatial media, focusing on distributed storage strategies, data integrity, and efficient retrieval at scale.

Problem Statement

Design a system to ingest and store millions of high-resolution, georeferenced street-view images captured by sensor arrays on fleet vehicles. Each vehicle uploads batches of image data along with telemetry (GPS, timestamp, orientation, device ID).

Functional Requirements

  • Ingestion: Provide an API for vehicles to securely upload imagery and metadata.
  • Storage: Persist raw imagery for long-term archival and processed imagery for public viewing.
  • Processing: Extract metadata and generate optimized thumbnails/tiles for fast serving.
  • Retrieval: Support spatial queries (e.g., "Get street-view at coordinate X, Y").

Non-Functional Requirements

  • Scale: Ingest ~50 TB of new data per day; maintain sub-second latency for tile retrieval.
  • Reliability: Data must be durable; uploads should support retries and resume-on-failure.
  • Security: Ensure authenticated uploads and integrity of source files.

Deliverables

  1. High-level architecture diagram (described in text).
  2. API design for ingestion and retrieval.
  3. Data model for storing imagery metadata and spatial indexing.
  4. Discussion of handling sudden traffic spikes from the fleet and ensuring eventual consistency.

Hints

Hint 1

Consider decoupling the upload process using a message queue to handle traffic bursts and ensure durability.

Hint 2

Think about using a spatial index like S2 cells or Quadtrees to map coordinates to image storage paths.

Hint 3

For massive scale, use object storage (e.g., S3/GCS) for images and a distributed database for metadata, separating the image binaries from the retrieval metadata.

Follow-up questions

  • How would you handle global consistency if the same street segment is captured by two different vehicles at different times?
  • What strategies would you implement to ensure data integrity during upload, such as handling partial uploads or corrupted image files?
  • How would you optimize the storage costs for the raw, high-resolution imagery versus the public-facing tile data?