Katkat

Implement streaming enrichment aggregator

BloombergSoftware Engineer · Technical Screen

Last updated

This problem tests your ability to design a stateful stream processor that aggregates fragmented event data based on a shared correlation ID within a sliding time window.

Problem Statement

You are building a real-time trade enrichment service. You receive two types of events from different upstream sources: OrderEvent and PriceEvent.

  • OrderEvent: Contains {order_id, symbol, quantity}.
  • PriceEvent: Contains {order_id, price}.

A complete record is formed only when both events for a specific order_id have arrived. Once both are received, you must output an EnrichedTrade object: {order_id, symbol, quantity, price}.

Implement an EnrichmentEngine class that processes these events as they arrive. If an event for an order_id arrives and its counterpart is missing, you must buffer the data in memory. To manage memory, implement a cleanup mechanism: if a partial record has been buffered for longer than T milliseconds, it is considered expired and should be discarded to free up memory.

Interface

class EnrichmentEngine:
    def __init__(self, timeout_ms: int):
        # timeout_ms is the duration a partial record stays in memory
        pass

    def process_order(self, order_id: str, symbol: str, quantity: int, timestamp_ms: int):
        pass

    def process_price(self, order_id: str, price: float, timestamp_ms: int):
        pass

Constraints

  • order_id is unique per trade.
  • Events can arrive out-of-order.
  • The system must be thread-safe.
  • Memory management is critical; you should be able to purge expired entries efficiently.

Example

  1. process_order("101", "AAPL", 10, 1000) -> No output, buffer: {"101": {symbol: "AAPL", qty: 10, ts: 1000}}.
  2. process_price("101", 150.0, 1050) -> Output: {"101", "AAPL", 10, 150.0}.
  3. If timeout_ms was 20 and a second OrderEvent arrived at 1040 without a price, the first entry would be evicted.

Hints

Hint 1

Use a hash map to store pending events keyed by the `order_id`.

Hint 2

Store events in a data structure that allows for easy retrieval and removal, such as a dictionary where values contain the partial record and arrival timestamp.

Hint 3

To handle expiration, consider using a secondary data structure like a sorted list or a priority queue, or perform a lazy cleanup during each new event processing.

Follow-up questions

  • How would you handle the case where multiple events of the same type arrive for the same ID (e.g., duplicate price updates)?
  • What changes would be necessary if the system needed to scale across multiple instances (distributed state)?
  • How would you measure the performance of this system, specifically regarding the latency between the arrival of the second event and the output?