Katkat
Back

Design a Time-Series Database

IllumioSoftware Engineer · Onsite
HardSoftware EngineerOnsiteSystem Design

Last updated

This problem tests the ability to architect a high-throughput, horizontally scalable system for time-series data, balancing ingest performance with efficient range-query capability.

Problem Statement

Design a distributed time-series database (TSDB) capable of handling 5 million write events per second with sub-second latency for range queries. The system must support labels (tags) for filtering series, such as host=server-1, region=us-east.

Functional Requirements

  • Write Path: Ingest metrics consisting of a timestamp, a set of key-value label pairs, and a double-precision value.
  • Read Path: Support range queries, such as "average CPU usage for region=us-east over the last hour."
  • Retention: Automatically expire data older than 30 days.

Non-Functional Requirements

  • High Availability: Must tolerate the loss of any single node without data loss.
  • Scalability: Must scale horizontally as the volume of metrics grows.
  • Consistency: Eventual consistency is acceptable for ingest, but query results should reflect the most recently committed data.

Deliverables

  1. Data Model: How do you store series metadata and time-indexed values to optimize for both high-write throughput and rapid range scans?
  2. Architecture: Describe the high-level components (e.g., ingestors, query engines, storage layers).
  3. Scaling Strategy: Explain how you would partition data to handle hotspots (e.g., a single hot server sending massive amounts of data) and how to manage index size given high label cardinality.
  4. Optimization: Propose a strategy for downsampling (e.g., converting 10s resolution to 1h resolution) and compaction to manage storage costs.

Hints

Hint 1

Consider the use of Log-Structured Merge (LSM) trees for the storage layer to handle high-write throughput while minimizing random I/O.

Hint 2

Think about how to separate the indexing of labels (an inverted index) from the actual time-series value storage to avoid excessive index fragmentation.

Hint 3

Address hotspots by using consistent hashing based on a combination of shard keys and time ranges, rather than just the metric name.

Follow-up questions

  • How would you handle 'late-arriving' data points that fall outside the current ingest window without corrupting compaction jobs?
  • If your query layer is struggling with high-cardinality labels (e.g., `user_id`), what techniques or restrictions would you implement to protect system stability?
  • How would you implement a distributed consensus mechanism to handle node discovery and membership, and why might you choose one protocol over another (e.g., Raft vs. Gossip)?