Katkat
Back

Design a Five-Minute Top-K Error Log Service

OracleSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

Design a fault-tolerant, high-throughput log aggregation service that identifies and ranks the top-K microservices with the highest error counts over a sliding five-minute window.

Scenario

A cloud platform hosts tens of thousands of microservices emitting high-volume telemetry. During incidents, site reliability engineers need a real-time dashboard displaying the top-K microservices experiencing the highest number of errors over a rolling 5-minute window. You need to design an end-to-end service to ingest, aggregate, and serve this data accurately and resiliently.

Requirements

Functional

  • Ingest structured error log records containing timestamp, service_id, error_code, and trace_id.
  • Provide an API getTopKServices(k, timestamp) that returns the top-K service_ids with their error counts over the trailing 5-minute window relative to the requested time.
  • Support late-arriving logs up to a defined threshold (e.g., 60 seconds late).
  • Support replaying logs from cold storage/message queues to rebuild state if needed.

Non-Functional

  • Latency: Dashboard queries must return in under 50 ms. Log-to-dashboard pipeline delay must be under 10 seconds.
  • Scalability: Must handle hot keys gracefully (e.g., a single malfunctioning service generating 80% of total error volume).
  • Accuracy: Exact error counts for top-K candidates are preferred over loose approximations.
  • Availability & Durability: Ingestion must not lose logs during node failures; query serving must remain available during aggregation worker crashes.

Scale & Assumptions

  • Total active microservices: 50,000.
  • Peak error log ingestion rate: 1,000,000 error events per second.
  • Average log size: 500 bytes (~500 MB/s ingestion bandwidth).
  • Query traffic: ~200 QPS (internal dashboards and automated alerts).
  • Default sliding window: 5 minutes, updated in increments of 10 seconds.

What Your Answer Should Cover

  • High-Level Architecture: Durable ingestion queue, stream aggregation layer, and serving store.
  • Windowing & Aggregation Model: Tumbling sub-buckets vs. continuous sliding windows, event-time processing, and watermark handling for late logs.
  • Hot Key & Skew Mitigation: Sharding strategies to prevent individual stream-worker bottlenecks.
  • Top-K Computation & Storage: Mechanics of merging sub-bucket counts and maintaining the top-K list.
  • Failure Modes: Broker failover, worker node crash recovery, and verification metrics to detect data discrepancies.

Hints

Hint 1

Deconstruct the 5-minute sliding window into discrete sub-buckets (such as 10-second tumbling windows) to allow incremental aggregation and reuse across overlapping windows.

Hint 2

Avoid sharding strictly by `service_id` at the raw ingestion layer; use two-stage aggregation (salting keys) to distribute the load of a hot service across multiple workers.

Hint 3

Use event-time watermarking to ensure late-arriving logs are incorporated into the appropriate sub-bucket before finalized top-K lists are published to the serving store.

Follow-up questions

  • How would you handle an extreme hot key where one service suddenly generates 90% of all platform errors without crashing downstream aggregation nodes?
  • If the number of unique services grows from 50,000 to 10,000,000, how would you transition from exact tracking to approximate streaming algorithms (such as Count-Min Sketch or HeavyKeepers)?
  • If a network partition causes aggregation workers to lag 15 minutes behind the ingestion stream, how does the system recover and backfill without causing stale dashboard reads?