Katkat
Back

Design a High-Concurrency Incident Ticket Correlation Platform

MicrosoftSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

Design a scalable incident ticket correlation and management system capable of handling high-volume alert storms, maintaining interval-based status history, and performing asynchronous, idempotent incident deduplication and linking.

Scenario

Large enterprise cloud platforms generate millions of operational alerts every day across thousands of microservices. During a major outage (e.g., widespread network degradation or regional datacenter failures), an alert storm can produce tens of thousands of duplicate or related tickets per second. You are tasked with designing a centralized Incident Ticket Correlation Platform (similar to Microsoft IcM) that ingests, deduplicates, correlates, and tracks the lifecycle of these incident tickets.

Requirements

Functional Requirements

  1. Ticket Ingestion & Linking: Ingest incident tickets from both automated monitoring monitors and human operators. Support explicit user-defined links and automated system links (e.g., parent-child relationships where hundreds of secondary alerts are grouped under a root cause incident).
  2. Interval-Based Status Tracking: Track incident lifecycles (e.g., Active, Mitigated, Resolved) using valid time intervals ([start_time, end_time)) to enable historical point-in-time state queries.
  3. Asynchronous Idempotent Correlation: Analyze incoming alerts asynchronously to link related incidents based on topology, service dependencies, and time windows. Ingestion and correlation operations must be idempotent.
  4. Search & Freshness: Support low-latency queries by service, severity, status, and time range. Query responses must include a "freshness indicator" showing how up-to-date the correlation state is.
  5. Reprocessing & Audit: Support retroactive reprocessing of correlation rules (e.g., if correlation heuristics are updated) and preserve an immutable audit trail of all manual and automated link decisions.
  6. Hot-Incident Resilience: Prevent cascading failures when a single "hot incident" receives 100,000+ incoming correlated events within a short window.

Non-Functional Requirements & Scale

  • Write Scale: Sustained 10,000 events/second; burst capacity of 100,000 events/second during alert storms.
  • Read Scale: 5,000 QPS for engineer dashboards and automated reporting.
  • Latency: Event ingestion acknowledgment p99 < 200 ms; correlation processing p99 < 5 seconds.
  • Availability & Durability: 99.99% availability; zero data loss for ingested incidents.

What to Cover

  • API Design: Core endpoints for ticket creation, status updates, manual/automated linking, and querying.
  • Data Model: Schemas for incidents, correlation relationships, interval-based status history, and audit records.
  • Architecture: Ingestion pipeline, message queues, correlation engine, and storage tiers.
  • Hot-Incident Mitigation: Strategies for handling lock contention, write amplification, and partition skew when an incident becomes "hot".
  • Idempotency & Reprocessing: Mechanics of ensuring safe retries, deduplication, and retroactive re-evaluation.

Hints

Hint 1

To handle write amplification on hot incidents, decouple the ingestion log from the parent ticket state by using append-only link events combined with periodic aggregation or local buffering.

Hint 2

For interval-based status tracking, avoid overwriting the current state; write a closed interval for the previous state and an open interval for the new state within a single atomic transaction.

Hint 3

Use deterministic idempotency keys derived from alert metadata (e.g., hash of service, environment, error signature, and time bucket) to prevent duplicate processing across retried messages.

Follow-up questions

  • How would you detect and resolve circular parent-child dependencies introduced by concurrent correlation workers?
  • If correlation rules are modified, how do you reprocess the last 7 days of incidents without taking down live ingestion or overwhelming downstream databases?
  • How does your query engine calculate and return the correlation freshness indicator without introducing high read overhead on the correlation queues?