Katkat
Back

Design a Telemetry Platform Ingesting Two Gigabytes per Second

FigureaiData Engineer · Technical Screen
MediumData EngineerTechnical ScreenSystem Design

Last updated

This question tests your ability to design a high-throughput, fault-tolerant telemetry ingestion pipeline capable of handling high-velocity data streams from distributed robotic systems.

Problem Statement

You are tasked with designing a telemetry platform for a fleet of robots. The system must ingest raw diagnostic and operational data at a sustained rate of 2 GB/s. The data consists of high-frequency sensor readings, state logs, and occasional image snapshots.

Requirements

  • Throughput: Support 2 GB/s ingress. Assume peak traffic may burst to 3 GB/s.
  • Durability: Zero data loss during ingestion. Ensure data is persisted before acknowledging receipt to the robots.
  • Scalability: The system must handle an increasing fleet size without manual intervention.
  • Processing: Support both real-time alerting (low latency) and long-term analytical storage (batch processing).
  • Schema Evolution: Handle frequent updates to robot sensor structures without breaking downstream pipelines.

Deliverables

Design a system architecture that covers:

  1. Data Flow: Explain how data travels from the robot edge to the final storage.
  2. Component Selection: Propose a technology stack (e.g., message brokers, processing frameworks, storage layers).
  3. Backpressure/Flow Control: How do you handle transient spikes or downstream system outages?
  4. Data Modeling: How do you store the data to allow for efficient querying while managing schema versions?

Scale Assumptions

  • Average event size: 1 KB.
  • Total daily volume: ~170 TB.
  • Retention: 30 days for hot storage, indefinite for cold storage.

Hints

Hint 1

Consider using a distributed message queue like Apache Kafka or Pulsar to act as a buffer between the edge devices and your processing layer.

Hint 2

Think about using a schema registry to manage versioning and validation as messages move through the pipeline.

Hint 3

For storage, evaluate the trade-offs between row-based formats (like Avro) for ingestion and column-based formats (like Parquet) for analytical queries.

Follow-up questions

  • How would you handle late-arriving data packets from robots that lose connectivity intermittently?
  • If one specific robot starts flooding the system with malformed data, how do you prevent it from impacting the rest of the fleet?
  • How would you implement a 'dead-letter queue' for data that fails schema validation, and what is your strategy for reprocessing it?