Katkat
Back

Explain Kafka and Message-Queue Delivery Semantics

ByteDanceSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSoftware Engineering Fundamentals

Last updated

This question tests your understanding of distributed messaging patterns, focusing on the architectural differences between log-based and queue-based systems and the trade-offs involved in ensuring reliable message delivery.

The Messaging Dilemma

Imagine you are building a distributed system where a producer service sends financial transaction events to a consumer service that updates a user's account balance. You are choosing between a traditional Message Queue (like RabbitMQ) and a Distributed Log (like Apache Kafka).

Task

  1. Compare the architecture and lifecycle of a message in a traditional work queue versus a log-based system like Kafka. How do these differences impact scalability and message persistence?

  2. Describe how you would implement 'at-least-once' delivery semantics in each system. Specifically, discuss how the consumer handles crashes during processing and how the system prevents data loss during producer retries.

  3. Suppose the consumer needs to perform non-idempotent updates (e.g., adding to a balance). If a consumer crashes after processing a message but before committing its offset/acknowledgment, it will re-process the message upon restart. Design a mechanism to ensure the final balance remains correct despite these duplicates.

Requirements for your response:

  • Explain the role of Partitions and Consumer Groups in Kafka for parallel processing.
  • Address how you maintain strict ordering of transactions for a specific user ID while still allowing high throughput across many users.
  • Discuss the trade-offs of using 'acks=all' in Kafka versus 'acks=1'.

Hints

Hint 1

Consider that a work queue deletes messages upon acknowledgment, while a log keeps data for a retention period regardless of consumption status.

Hint 2

For idempotency, think about tracking 'processed' identifiers or using versioned updates that ignore messages with a sequence number lower than the one currently recorded.

Hint 3

To maintain per-user ordering in a distributed system, ensure all events for a specific user ID are routed to the same partition.

Follow-up questions

  • How does 'Exactly-Once Semantics' (EOS) work in Kafka, and what are the performance overheads of using Kafka transactions?
  • If your consumer performs a database write and then an offset commit, how can you make these two operations atomic?
  • Under what scenarios might a consumer read the same message twice, even without a crash?