Katkat
Back

Design an Internal Metrics Ingestion Service

OracleSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

Design a scalable internal metrics collection and storage platform capable of ingesting high-throughput time-series data from hundreds of thousands of hosts, decoupling ingestion from query workloads.

Scenario

Internal cloud infrastructure teams require a robust, centralized metrics collection service to monitor thousands of microservices and hardware hosts. The platform must collect operational metrics (e.g., CPU, memory, request latencies, error counts), store them durably, and support dashboarding and alerting queries.

Your task is to design this metrics ingestion and querying service, focusing on agent design, high-throughput write path, data partitioning, and isolation between writes and read queries.

Requirements

Functional Requirements

  • Metric Ingestion: Accept time-series data points containing a metric name, timestamps, key-value dimensions/tags, and numeric values.
  • Client Reporting: Provide a lightweight mechanism on host machines to report metrics reliably without impacting host workloads.
  • Querying: Support time-range queries, downsampling, and aggregations (e.g., avg, p99, sum) filtered and grouped by metric tags.
  • Data Retention & Downsampling: Retain high-resolution raw data for short-term operational visibility, followed by aggregated historical rollups for long-term trends.

Non-Functional Requirements

  • High Availability: Ingestion availability must be 99.99%. Dropped metrics should be minimized even during regional network hiccups.
  • Write-Heavy Architecture: Optimize primarily for writes while ensuring heavy user dashboard queries cannot degrade write throughput.
  • Horizontal Scalability: Support linear scale-out as the number of hosts and services grows.

Scale & Performance Assumptions

  • Fleet Size: 100,000 host nodes / containers.
  • Ingestion Volume: Each host reports ~50 distinct metrics every 10 seconds (~500,000 data points/sec average, peaking at 1,000,000 points/sec).
  • Retention: Raw data retained for 7 days; 1-minute rollups for 30 days; 1-hour rollups for 1 year.
  • Query Latency: P95 query response time < 500ms for dashboards over the last 6 hours.

Suggested Areas to Address

  1. Client Agent Architecture: Push vs. pull models, local batching, in-memory ring buffers, and backpressure handling on the host.
  2. Ingestion & Storage Architecture: Ingestion load balancing, distributed streaming/buffering (e.g., Kafka), and specialized time-series storage engines (LSM-trees, chunking, columnar storage).
  3. Sharding & Partitioning: Strategy for distributing series across shards (e.g., hashing metric_name + tags vs. time-window sharding).
  4. Workload Separation: Architectural decoupling of high-volume writes from analytical read/query engines.

Hints

Hint 1

Consider buffering metrics locally on the agent using a ring buffer to smooth out network blips and reduce per-request HTTP overhead.

Hint 2

Hash partitioning by metric identity (series key = metric name + sorted tag set) avoids hotspotting single nodes, but combine it with time-bucketed storage to simplify old data expiration.

Hint 3

Separate the read and write paths by decoupling ingestion via a distributed log and maintaining separate read-optimized replicas or downsampled rollups.

Follow-up questions

  • How would you detect and mitigate high-cardinality explosions caused by engineers adding unique identifiers (like user IDs or transaction IDs) to metric tags?
  • If an ingestion broker cluster experiences an outage for 10 minutes, how do agents handle the backlog without crashing host memory or causing a thundering herd on recovery?
  • How would you implement background downsampling and retention rollups without causing I/O contention on active ingestion writes?