Katkat
Back

Explain Kafka and Design Bulk Record Processing

BalyasnySoftware Engineer · Onsite
HardSoftware EngineerOnsiteSystem Design

Last updated

This problem tests knowledge of distributed event streaming architecture, specifically handling high-throughput bulk processing while maintaining partition-level ordering and fault-tolerant delivery guarantees.

Scenario

You are designing a financial trade-processing pipeline that consumes trade events from a high-volume Kafka topic. Your system must perform bulk calculations (e.g., aggregating risk exposure over 1-minute windows) and persist results to a downstream database.

Requirements

  1. Ordering: Trade events for a specific ticker_id must be processed in the exact order they appear in the partition.
  2. Batching: Aggregations should be triggered every 1000 records or every 5 seconds, whichever comes first.
  3. Backpressure: The system must gracefully handle downstream database slowdowns without crashing the consumer or losing data.
  4. Fault Tolerance: If a processing node fails, the system must support replaying from the last successfully committed offset.
  5. Exactly-Once Semantics: Ensure that even in the event of failures, trade aggregates are not written to the database more than once.

Task

Design a system architecture to satisfy these requirements. Your design should include:

  • Kafka consumer group configuration strategies (partitioning, rebalancing).
  • The buffer mechanism for batching and how you manage windowing.
  • A strategy for managing database transactions alongside Kafka offset commits.
  • How to handle backpressure when the database throughput drops.

Constraints

  • Assume 50k events per second across 20 partitions.
  • Downstream latency to the database is variable (10ms to 500ms).

Hints

Hint 1

Consider the trade-off between committing offsets before or after database writes and how an atomic transaction or idempotent write can enforce exactly-once guarantees.

Hint 2

To handle backpressure, consider using a non-blocking queue between the consumer and the aggregator, or pausing the Kafka consumer via the poll loop when local buffers reach capacity.

Hint 3

Since order is required per ticker, ensure that your partitioning strategy maps the same ticker IDs to the same consumer instances.

Follow-up questions

  • How would you handle a 'poison pill' message that causes the processing logic to crash repeatedly?
  • If you needed to reprocess the last 24 hours of data for a system bug fix, how would you design the system to handle the load spike without affecting real-time traffic?
  • How does Kafka's replication factor and ISR (In-Sync Replica) set impact your data loss guarantees during a broker failure?