Katkat
Back

Design streaming mention analytics with search and alerts

BloombergSoftware Engineer · Onsite
HardSoftware EngineerOnsiteSystem Design

Last updated

This question tests the ability to design a distributed, low-latency streaming pipeline that aggregates mention counts for entities, provides search capabilities, and triggers alerts based on configurable thresholds.

Scenario

Design a streaming analytics platform for a financial news terminal that processes 100,000 mentions per second across global financial markets. Users want to track the frequency of entity mentions (e.g., ticker symbols like AAPL or TSLA) in real-time, search historical mention data, and receive instant alerts when a specific entity's mention volume exceeds a defined threshold within a 5-minute sliding window.

Requirements

Functional Requirements

  • Ingestion: Consume mention events (timestamp, entity_id, source, content_snippet).
  • Aggregation: Maintain running mention counts per entity for sliding time windows (e.g., 5m, 1h, 24h).
  • Search: Support querying historical mention data by entity_id and time_range.
  • Alerting: Trigger notifications when an entity exceeds a user-defined threshold within a sliding window.

Non-Functional Requirements

  • Latency: End-to-end latency for alerts should be under 5 seconds.
  • Scalability: The system must handle bursts of traffic without losing data.
  • Consistency: Eventual consistency is acceptable for analytics, but alerts must be triggered reliably.

Task

Provide a high-level architecture diagram and describe the following components:

  1. Data Ingestion: How will you ingest and buffer high-volume event streams?
  2. Stream Processing: Describe your stateful aggregation strategy and how you handle late-arriving data.
  3. Storage & Search: Define your storage layer strategy for both hot (real-time/sliding window) and cold (historical) data.
  4. Alerting Service: Explain how the threshold monitoring and notification dispatching logic will work.

Considerations

  • How do you handle "hot" entities (e.g., a major market-moving news event causing a massive spike in mentions)?
  • How will you manage the retention and partitioning of the historical data?

Hints

Hint 1

Consider using a distributed streaming platform like Apache Kafka to decouple ingestion from processing and provide back-pressure handling.

Hint 2

Think about using a two-tier storage architecture: an in-memory key-value store (e.g., Redis) for rapid windowed aggregations and a distributed search engine (e.g., Elasticsearch) for long-term historical querying.

Hint 3

To address hot entities, use consistent hashing or key-based partitioning to distribute the load across multiple stream processor instances.

Follow-up questions

  • How would you handle "exactly-once" processing semantics if the alert trigger logic requires avoiding duplicate notifications?
  • If the historical search functionality needs to support sub-second response times for queries spanning 30 days of data, how would you optimize your storage schema?
  • How would you modify this system if users requested the ability to see the 'sentiment' of mentions (e.g., positive vs. negative) in addition to volume counts?