Design a Resilient Chat System

Scenario

Design the backend infrastructure for a resilient enterprise chat platform supporting both 1:1 conversations and medium-to-large group channels. The system must guarantee that once a sender receives an acknowledgment, the message is never lost—even if individual nodes or in-memory queues crash. Furthermore, recipients must observe messages in a strictly deterministic order and seamlessly catch up after network disconnections without missing or duplicating messages.

Requirements

Functional Requirements

• 1:1 and Group Chat: Support direct messaging between two users and group conversations with up to 1,000 participants.
• Message Delivery & Sync: Real-time push delivery to connected clients; catch-up synchronization for offline clients upon reconnection.
• Delivery Guarantees & Idempotency: At-least-once transport with client-side deduplication (end-to-end exactly-once semantics from the user's perspective).
• Group Membership Changes: Enforce dynamic membership updates so users can only read messages sent during their active membership window.

Non-Functional Requirements

• Durability: Messages must be committed to durable storage before an acknowledgment is returned to the sender. Acknowledging data residing solely in volatile broker memory is unacceptable.
• Ordering: Strict per-conversation causal/monotonic ordering.
• Latency: P99 delivery latency under 100 ms for active connections.
• High Availability: Resilient to gateway or broker failures without data corruption or permanent message loss.

Scale & Numbers

• Daily Active Users (DAU): 50 million.
• Messages per Day: 1 billion messages (~12,000 writes/sec average, 50,000 writes/sec peak).
• Concurrent Active Connections: 10 million simultaneous WebSocket/gRPC streams.
• Group Size: Median group size is 8; 99th percentile is 200; hard limit is 1,000.

Key Areas to Address

1. API & Interface Design: Message sending, real-time streaming, and catch-up sync endpoints.
2. Data Storage & Ordering: How sequence numbers are assigned, data schemas for messages and conversation states, and partition strategies.
3. Message Lifecycle & Fan-out: End-to-end tracing from client publish -> gateway -> sequencer/persistence -> fan-out engine -> push gateways.
4. Durability vs. Volatile Broker Risks: Explain the trade-offs of using memory-only message brokers (e.g., Redis Pub/Sub) vs. write-ahead-log (WAL) backed systems (e.g., Kafka, ScyllaDB/Cassandra) before sender ACK.
5. Reconnection & Reconciliation: Protocol for clients reconnecting with a last_delivered_sequence_id after network drops or node failovers.