Katkat
Back

Design a Data Stream Processor

AtlassianSoftware Engineer · Technical Screen
EasySoftware EngineerTechnical ScreenSystem Design

Last updated

This question tests your ability to design a scalable, reliable stream processing pipeline capable of ingesting, filtering, and persisting real-time events while managing system state.

Design a Data Stream Processor

Design a service that processes a high-volume stream of incoming events (e.g., user activity logs) from multiple clients. The system must perform the following tasks:

  1. Ingestion: Accept incoming JSON events via an HTTP API.
  2. Validation & Filtering: Drop events that are malformed or do not meet specific schema requirements.
  3. Deduplication: Ensure that if the same event_id is received twice within a 5-minute window, the duplicate is discarded.
  4. Persistence: Store valid events in a data store for long-term analytical queries.

Requirements & Constraints

  • Scale: The system should handle 10,000 events per second.
  • Latency: End-to-end processing latency should be under 500ms.
  • Reliability: Minimal data loss is expected during system updates or minor component failures.
  • Storage: Must support high-write throughput and time-based range queries.

Expectations

Provide a high-level architecture diagram or description. Specifically address:

  • How will you handle the deduplication window at this scale?
  • What partitioning strategy would you use for the message queue or database?
  • How does the system handle backpressure if the downstream consumer is slow?
  • What metrics would you monitor to ensure system health?

Hints

Hint 1

Consider using an intermediary distributed message broker like Kafka to decouple ingestion from processing.

Hint 2

For deduplication, think about using a fast, time-to-live (TTL) enabled key-value store like Redis to track recent event IDs.

Hint 3

Evaluate the trade-off between strict ordering of events versus higher parallelism and throughput.

Follow-up questions

  • How would you handle a scenario where a downstream database becomes unavailable for several minutes?
  • If the deduplication window needs to increase from 5 minutes to 24 hours, how would your architecture change?
  • How would you implement 'exactly-once' processing semantics if required by the business?