Katkat
Back

Design an exception monitoring system with top‑K

LinkedInSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

This question tests the ability to design a high-throughput distributed system capable of ingesting, deduplicating, and aggregating exception logs to provide real-time top-K statistics.

Problem Statement

Design an Exception Monitoring System that collects error events from thousands of microservices and displays a real-time dashboard showing the 'Top 10 most frequent exceptions' over the last hour.

Functional Requirements

  1. Ingest millions of exception events per second.
  2. Deduplicate similar exceptions using a fingerprinting mechanism.
  3. Maintain a rolling 'Top-K' list of exception frequencies for the last 60 minutes.
  4. Provide an API to query the frequency of specific exceptions.

Non-Functional Requirements

  1. High availability and fault tolerance.
  2. Low latency for ingestion (asynchronous processing is acceptable).
  3. Data privacy: Ensure PII (Personally Identifiable Information) in stack traces is masked before storage.
  4. Scalability: The system must handle bursts of traffic without losing data.

Expected Deliverables

  • Define the data model for raw logs and aggregated exception counts.
  • Explain the ingestion pipeline, including the strategy for grouping similar exceptions into a unique fingerprint.
  • Propose an architectural approach for calculating the Top-K in a streaming environment.
  • Describe the storage strategy for long-term retention vs. real-time query performance.

Hints

Hint 1

Consider using a fingerprinting function that strips variable identifiers (like memory addresses or IDs) from stack traces to group similar errors.

Hint 2

Think about using a message queue (like Kafka) to decouple ingestion from processing and a time-series database or Redis for the real-time aggregations.

Hint 3

For the Top-K requirement, evaluate if you need an exact count or if an approximation algorithm like Count-Min Sketch or Heavy Hitters would be more efficient at scale.

Follow-up questions

  • How would you handle a sudden 'exception storm' where one service starts logging thousands of errors per second, potentially overloading the system?
  • What architectural changes are needed if we want to alert on-call engineers when the frequency of a specific exception increases by 50% within a 5-minute window?
  • How would you ensure that PII masking is applied consistently across all services while minimizing the performance impact on the calling application?