Katkat
Back

Explain Backend Infrastructure Fundamentals

ByteDanceSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSoftware Engineering Fundamentals

Last updated

This question tests your understanding of Kafka's distributed messaging architecture, specifically how message distribution, ordering guarantees, and consumer scalability interact during consumer group operations.

The Kafka Consumer Group Lifecycle

Imagine you are designing a high-throughput logging system using Apache Kafka. You have a single topic with 10 partitions and a consumer group consisting of 4 consumer instances reading from this topic.

Your Tasks

  1. Message Distribution: Explain how Kafka decides which consumer receives a message from a specific partition. What is the impact if the number of partitions (10) is not evenly divisible by the number of consumers (4)?

  2. Ordering Guarantees: Describe the extent to which Kafka guarantees message ordering. Under what circumstances can this order be violated at the consumer level, and how would you enforce strict global ordering if required?

  3. Rebalancing: Explain the mechanism of a 'rebalance' when a consumer instance crashes. What happens to the offsets, and how does the cluster decide which consumer takes over the abandoned partitions?

Example Scenario

Suppose you have 10 partitions (P0 through P9). If you assign them to 4 consumers (C1 through C4), explain the distribution logic. If C4 suddenly disconnects, describe the lifecycle of the transition until the system stabilizes.

Hints

Hint 1

Recall that Kafka consumer groups assign partitions to instances, not individual messages, to ensure a single partition is processed by exactly one consumer in the group at a time.

Hint 2

Consider that message ordering is guaranteed only within a single partition, and any cross-partition ordering requires application-level coordination or a single-partition constraint.

Hint 3

Think about the role of the Group Coordinator and the Consumer Group Leader in executing the partition rebalance protocol and committing offsets to the internal `__consumer_offsets` topic.

Follow-up questions

  • How would you handle a scenario where processing a single message takes significantly longer than others, leading to consumer lag?
  • What are the trade-offs between 'at-least-once' and 'exactly-once' delivery semantics in the context of Kafka consumer offsets?
  • How does the choice of a partition strategy (e.g., Round Robin vs. Range vs. Sticky) impact the performance and stability of a consumer group during rebalances?