Katkat
Back

Design Chat Deletion Semantics Under Concurrent Sends

DatabricksSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSystem Design

Last updated

Design the data consistency, concurrency control, and deletion semantics for a distributed chat platform when message creation and thread deletion occur concurrently across distributed nodes.

Scenario

In an enterprise collaboration platform, users communicate in persistent discussion threads. A common distributed concurrency bug occurs when User A deletes a thread at the exact moment User B submits a new message into that thread across different edge servers. Without well-defined semantics, the system may create orphaned "zombie" messages that re-open the deleted thread or cause state divergence between search indexes, caches, and client devices.

Design the end-to-end deletion and messaging semantics to ensure correct behavior under high concurrency and distributed network delays.

Requirements

Functional Requirements

  • Send Message: Users can post a message into an existing thread with retry support (idempotency).
  • Delete Thread: A user or moderator can delete an entire thread.
  • Read / Sync: Online and reconnecting offline clients must sync thread state and message history without observing ghost updates.
  • Audit & Compliance: Deleted threads must adhere to compliance retention (e.g., retained internally for legal hold / audit, but strictly hidden from normal user operations).

Non-Functional Requirements

  • Consistency & Linearization: Establish a clear visible contract: once a thread deletion is committed, no subsequent or concurrent message should ever leave the thread in an active/resurrected state.
  • Low Latency: Message send p99 latency < 100ms.
  • Scale: 50 million daily active users (DAU), 1 billion messages per day, peak write throughput of 40,000 messages/sec, and 5 million thread deletions per day.

What to Cover

  1. Consistency Contract & Concurrency Control: Define the ordering rules (e.g., monotonic versioning, logical clocks, or DB transactions). Explain how conditional writes (CAS / transactions) and tombstones prevent orphaned messages.
  2. Data Model: Storage schema for threads, messages, and tombstones.
  3. Propagation & Invalidation: How deletion events propagate across distributed caches (e.g., Redis), pub/sub channels, and search indexes (e.g., Elasticsearch).
  4. Offline Client Synchronization: How an offline client reconciling pending local drafts avoids resurrecting deleted threads.
  5. Audit Retention vs. User Visibility: How soft-deletion and hard-purge lifecycles interact with legal hold.

Hints

Hint 1

Consider modeling thread state with monotonic version numbers or epoch timestamps so message writes can execute a Compare-And-Swap (CAS) asserting the thread is active.

Hint 2

Think about the tombstone lifecycle: dropping a thread row immediately makes it impossible to distinguish between a non-existent thread and a recently deleted thread during concurrent writes.

Hint 3

For offline clients attempting to send messages after reconnecting, the sync protocol needs to explicitly reject messages aimed at tombstoned thread IDs rather than blindly inserting them.

Follow-up questions

  • How would you handle a race condition where a message batch write starts before thread deletion begins but commits after the deletion transaction finishes?
  • If hard-deleting records for GDPR/compliance is legally required within 30 days, how do you safely clean up tombstones without re-introducing race conditions for delayed network packets?
  • How does your architecture scale if a popular public channel with 500,000 members is deleted while receiving hundreds of messages per second?