Katkat
Back

Design an Accurate Click Aggregator

AppleSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSystem Design

Last updated

This question tests the ability to design a high-throughput, fault-tolerant system for tracking ad clicks while ensuring exactly-once processing or deduplication at scale.

Scenario

You are asked to design a click-tracking system for a global advertising platform. The system must process millions of clicks per second with sub-second latency for real-time dashboards and ensure that no click is double-counted, even in the event of retries from clients or infrastructure failure.

Requirements

  1. Functional:

    • Track clicks for specific ad IDs.
    • Provide an API to query the total click count for a given ad ID in the last hour.
    • Ensure high durability and accuracy (deduplication).

  2. Non-functional:

    • Scale: Handle 10 million clicks per second.
    • Latency: The ingestion path must be highly performant.
    • Accuracy: Deduplication is mandatory (same user clicking the same ad within a short time window).

What your answer should cover

  • API Design: Define the interface for the client-side event ingestion.
  • Data Flow: Describe the path from the click event ingestion to the final aggregation storage.
  • Deduplication Strategy: How do you handle duplicate requests? Consider the trade-offs between local/distributed state and storage backends.
  • Scalability and Fault Tolerance: How do you partition data to handle the load and ensure system availability?

Hints

Hint 1

Consider using a unique event ID or a composite key (User_ID, Ad_ID, Timestamp_Window) to identify duplicate clicks.

Hint 2

A distributed log like Kafka can provide a durable buffer, but consider how consumer groups or stateful processing frameworks (e.g., Flink) help with aggregation.

Hint 3

Think about the trade-off between using a high-speed distributed cache (like Redis) for deduplication versus a persistent database, considering latency requirements.

Follow-up questions

  • How would your architecture change if we needed to calculate clicks over a 24-hour window instead of a one-hour window?
  • What happens if the deduplication storage layer becomes a bottleneck? How would you shard the data?
  • How would you ensure data consistency if the system requires 'exactly-once' processing semantics versus 'at-least-once'?