Design an ad-click aggregation and enrichment pipeline

Scenario

You are designing a data platform to measure advertising performance.

Mobile apps and web browsers send ad impression and ad click events. Analysts need near-real-time dashboards and batch reports.

Requirements

• Ingest impression/click events from mobile and web clients.
• Produce aggregates such as:
• clicks / impressions / CTR
• grouped by time window (e.g., 1 min, 1 hour, 1 day)
• grouped by dimensions like campaign_id, ad_id, publisher_id, country, device_type
• Enrich events by joining with other data sources (examples):
• campaign metadata (budget, objective)
• ad metadata (creative type)
• user/device attributes (coarse geo, OS)
• Support both:
• near-real-time queries (seconds to a few minutes delay)
• historical queries over months
• Event delivery constraints:
• clients may be offline and retry
• duplicate/out-of-order events can occur

Scale & SLOs (assume)

• Peak 500k events/sec (impressions+clicks), average 100k/sec.
• Dashboard freshness: P95 < 2 minutes.
• Correctness: exactly-once is not required, but duplicates should be minimized and results should be explainable.

Key discussion prompt

Clients can send events:

1. one request per event, or
2. batch multiple events per request.

Explain the trade-offs between number of requests vs latency, especially for mobile networks.

Deliverables

• High-level architecture and major components
• Data model / schemas
• How you do enrichment joins (stream-stream vs stream-table vs batch)
• How you handle deduplication, late events, and backfills
• What you store for serving (OLAP/warehouse) and for near-real-time dashboards