Katkat
Back

Reason About Kafka Consumers and Partitions

Wells FargoSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSoftware Engineering Fundamentals

Last updated

This question tests your understanding of Kafka's consumer group protocol, partition assignment logic, and the mechanics of horizontal scaling in distributed message queues.

The Scenario

You are designing a high-throughput transaction processing system using Apache Kafka. Your topic is configured with 3 partitions to store incoming financial events. Currently, a consumer group is processing these events.

The Problem

  1. Suppose you have a consumer group with 5 active consumers assigned to consume from this topic (3 partitions). How will Kafka assign the partitions to the consumers, and what will the status of the remaining 2 consumers be?
  2. Explain the concept of the consumer offset. If a consumer crashes after processing a message but before committing the offset, what happens to that message upon a rebalance? How can you ensure at-least-once processing semantics in this scenario?
  3. You are asked to increase throughput by adding more consumers. Is adding more consumers than partitions an effective way to scale this system? Justify your answer based on Kafka's threading and partition ownership model.
  4. Describe the sequence of events during a partition rebalance. What impact does this have on consumer latency and system availability during the transition period?

Hints

Hint 1

Recall that Kafka enforces a one-to-one mapping where a single partition can be read by only one consumer within a group at any given time.

Hint 2

Consider the difference between auto-commit and manual commit strategies when handling potential consumer failures.

Hint 3

Analyze the bottleneck: if you have 10 consumers but only 3 partitions, consider how many consumers are actually performing work versus sitting idle.

Follow-up questions

  • How would you reconfigure the system if you needed to double the current processing throughput?
  • What are the trade-offs between using a 'range' partition assignment strategy versus a 'sticky' partition assignment strategy?
  • How does the concept of a 'consumer group coordinator' facilitate the detection of a dead consumer?