Katkat
Back

Design a Reliable Notification System

MetaSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

Design an enterprise-scale notification platform supporting multi-channel delivery, recipient fanout, preference filtering, and resilient at-least-once delivery guarantees.

Scenario

Design a unified notification service for a large-scale social platform. The system must accept notification triggers from various internal microservices (e.g., post likes, direct messages, friend requests, system announcements) and reliably deliver them to users across multiple channels (iOS/Android push notifications, SMS, email, and in-app notification center).

Requirements

Functional Requirements

  • Multi-channel delivery: Route messages to push providers (APNs, FCM), SMS aggregators, email gateways, and the in-app inbox.
  • User preferences and deduplication: Respect user settings (channel opt-outs, quiet hours / Do Not Disturb) and aggregate rapid duplicate triggers (e.g., "Alice and 14 others liked your photo").
  • Recipient fanout: Support both 1-to-1 events (direct message) and 1-to-many fanout (a creator with 20 million followers posting a new video).
  • In-app notification storage: Allow users to fetch paginated notification history and mark items as read.

Non-Functional Requirements & Scale

  • Scale: 10 billion notifications generated per day (~120,000 requests/sec average, 500,000/sec peak).
  • Latency: Critical alerts (e.g., 2FA, DMs) must be delivered to external gateways within 2 seconds. Bulk notifications (e.g., digest updates) can take up to several minutes.
  • Reliability: At-least-once delivery with zero data loss on ingestion. Gracefully handle third-party gateway downtime without cascading backpressure.
  • Idempotency: Prevent duplicate alerts to the end-user device when upstream services retry.

What Your Design Should Cover

  1. API Design: Endpoints for internal services to trigger notifications and client endpoints to read inbox items.
  2. High-Level Architecture: Ingestion pipelines, queue topologies, worker pools, preference checking, and gateway dispatchers.
  3. Data Model & Storage: Schemas for inbox storage, user delivery preferences, and delivery status tracking.
  4. Failure Modes & Trade-offs: Retry strategies, dead-letter queues, handling slow third-party providers, and fanout strategies for high-follower accounts.

Hints

Hint 1

Decouple ingestion from third-party gateway dispatch using tiered priority queues so slow external APIs do not block high-priority transactional alerts.

Hint 2

For broadcast fanout to millions of followers, consider a hybrid push-pull approach or asynchronous background chunking rather than immediate full expansion at ingestion.

Hint 3

Ensure deduplication and idempotency keys are tracked at both the ingestion layer and the external dispatch worker layer to avoid duplicate push alerts.

Follow-up questions

  • How would you implement frequency capping and notification collapse (e.g., max 3 notifications per hour per category, or bundling 50 likes into one notification)?
  • How does your system distinguish between transient third-party errors (e.g., rate limits, 503s) and permanent failures (e.g., invalid device tokens), and how are stale tokens cleaned up?
  • How would you partition the in-app notification inbox database to maintain low-latency reads for unread counts across hundreds of millions of daily active users?