Design a Scalable Notification Service

Scenario

Salesforce powers critical communication workflows for millions of enterprises—ranging from urgent security alerts and transactional updates (e.g., password resets, order confirmations) to automated marketing digests. You are tasked with designing a centralized, highly reliable Notification Service capable of dispatching notifications across multiple channels (Email, SMS, Mobile Push, and In-App).

Functional Requirements

• Multi-Channel Ingestion: Accept notification requests targeting multiple recipients across various channels (Email, SMS, Push, In-App/Webhook) via a unified API.
• Explainable Filtering & Preferences: Evaluate user communication preferences (opt-outs, channel priority, quiet hours, and frequency capping). Maintain an audit trail explaining why any notification was sent, altered, or suppressed.
• Quota & Rate Limiting: Enforce per-tenant and per-recipient rate limits to prevent spam, avoid noisy neighbors, and protect downstream provider quotas.
• Guaranteed & Retry-Safe Delivery: Ensure at-least-once delivery for critical alerts with end-to-end idempotency, exponential backoff retries, and dead-letter queues (DLQ).
• Extensible Provider Adapters: Provide an abstraction layer to integrate external delivery vendors (e.g., Twilio, SendGrid, APNs, FCM) with dynamic failover capabilities.

Scale & Non-Functional Requirements

• Traffic Scale: 200 million notifications per day (~2,300 notifications/sec average, with bursts up to 25,000 notifications/sec during marketing campaigns or platform incidents).
• Latency: Critical alerts (e.g., 2FA/OTPs) must deliver within 3 seconds (p99); bulk/marketing notifications can tolerate delivery within 15 minutes.
• Reliability: 99.99% availability with zero message loss for transactional messages.
• Idempotency: Prevent duplicate sends caused by network retries from upstream clients or internal worker restarts.

Key Areas to Address

1. API & Data Model: Design the ingestion API schema and key data entities (templates, recipient preferences, delivery tasks, audit logs).
2. High-Level Architecture: Diagram the flow from ingestion to routing, preference checking, rate limiting, and dispatch.
3. Queueing & Priority Management: Explain your partitioning strategy, priority queues (critical vs. bulk), and backpressure handling.
4. Failure Modes & Explainability: Detail how the service handles vendor outages, circuit breaking, idempotency keys, and structured audit logs.