Katkat
Back

Design an Alert Notification Service for Downstream Consumers

DoorDashSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSystem Design

Last updated

Design a reliable, scalable alert and event notification delivery platform that decouples upstream producers from heterogeneous downstream consumers with independent fan-out, isolation, retries, and dead-letter handling.

Scenario

At an on-demand logistics platform like DoorDash, real-time events (e.g., OrderPlaced, DasherArrived, StoreClosed) must be distributed to dozens of internal services and external partner webhooks. Each downstream subscriber has different throughput limits, reliability profiles, and payload version requirements. You need to design an Alert Notification Service that ingests high-volume producer events and delivers them independently and reliably to registered consumers.

Requirements

Functional Requirements

  1. Event Ingestion & Routing: Ingest events via an API and route them to matching downstream subscribers based on subscription criteria and payload schema versions.
  2. Independent Delivery & Isolation: Delivering an event to a slow or failing consumer must not block or delay delivery to any other consumer.
  3. Reliability & Duplicate Handling: Guarantee at-least-once delivery. Detect and deduplicate redundant producer events using idempotency keys.
  4. Classified Retries & Dead Letter Queue (DLQ): Differentiate transient failures (e.g., HTTP 503, network timeout) from non-retryable errors (e.g., HTTP 400, 404). Retry transient errors with exponential backoff and route unrecoverable events to a DLQ.
  5. Replay & Auditability: Allow operators to replay events for a specific consumer over a designated time window. Persist an audit trail of every delivery attempt and status.

Non-Functional Requirements

  1. High Throughput & Low Latency: Ingest up to 50,000 events/second at peak. Healthy subscribers should receive events within 1 second (p99).
  2. Fault Tolerance & Backpressure: Apply rate limits and backpressure per consumer endpoint to avoid overwhelming external systems or internal services.
  3. Data Retention: Retain raw events for at least 7 days to support debugging and replays.

Scale Assumptions

  • Peak Event Ingestion: 50,000 events/second.
  • Downstream Consumers: ~500 registered consumer configurations.
  • Fan-Out Factor: Average of 5 subscribers per event (~250,000 deliveries/second peak).
  • Audit Log Volume: Up to 300M delivery attempt records per day.

What Your Answer Should Cover

  • API Design: Ingestion endpoint, subscription management, and replay trigger.
  • Data Model: Storage schema for event storage, subscriptions, delivery logs, and idempotency tracking.
  • System Architecture: Ingestion tier, fan-out engine, worker dispatchers, retry queues, and DLQ handling.
  • Failure Modes & Trade-offs: Head-of-line blocking mitigation, deduplication windows, circuit breaking, and replay mechanics.

Hints

Hint 1

Decouple the raw event ingestion log from per-consumer delivery queues to prevent a slow downstream endpoint from stalling the entire pipeline.

Hint 2

Use an in-memory key-value store with TTLs (e.g., Redis) for fast idempotency checks at ingestion, backed by an immutable append-only event store for audit and replay.

Hint 3

Implement categorized retry queues (e.g., 1m, 5m, 15m delay topics) combined with consumer-level circuit breakers to avoid overwhelming degrading downstream endpoints.

Follow-up questions

  • How would you handle a downstream consumer that suffers extended downtime without exhausting delivery queue storage or dropping messages?
  • How does your schema versioning strategy handle breaking changes in event payloads without breaking older consumers that have not upgraded?
  • If an operator triggers a replay of 10 million events for a single consumer, how do you prevent that replay traffic from starving real-time deliveries for other consumers?