Implement streaming enrichment aggregator
BloombergSoftware Engineer · Technical ScreenLast updated
This problem tests your ability to design a stateful stream processor that aggregates fragmented event data based on a shared correlation ID within a sliding time window.
Problem Statement
You are building a real-time trade enrichment service. You receive two types of events from different upstream sources: OrderEvent and PriceEvent.
OrderEvent: Contains{order_id, symbol, quantity}.PriceEvent: Contains{order_id, price}.
A complete record is formed only when both events for a specific order_id have arrived. Once both are received, you must output an EnrichedTrade object: {order_id, symbol, quantity, price}.
Implement an EnrichmentEngine class that processes these events as they arrive. If an event for an order_id arrives and its counterpart is missing, you must buffer the data in memory. To manage memory, implement a cleanup mechanism: if a partial record has been buffered for longer than T milliseconds, it is considered expired and should be discarded to free up memory.
Interface
class EnrichmentEngine:
def __init__(self, timeout_ms: int):
# timeout_ms is the duration a partial record stays in memory
pass
def process_order(self, order_id: str, symbol: str, quantity: int, timestamp_ms: int):
pass
def process_price(self, order_id: str, price: float, timestamp_ms: int):
pass
Constraints
order_idis unique per trade.- Events can arrive out-of-order.
- The system must be thread-safe.
- Memory management is critical; you should be able to purge expired entries efficiently.
Example
process_order("101", "AAPL", 10, 1000)-> No output, buffer:{"101": {symbol: "AAPL", qty: 10, ts: 1000}}.process_price("101", 150.0, 1050)-> Output:{"101", "AAPL", 10, 150.0}.- If
timeout_mswas 20 and a secondOrderEventarrived at1040without a price, the first entry would be evicted.
Hints
Hint 1
Use a hash map to store pending events keyed by the `order_id`.
Hint 2
Store events in a data structure that allows for easy retrieval and removal, such as a dictionary where values contain the partial record and arrival timestamp.
Hint 3
To handle expiration, consider using a secondary data structure like a sorted list or a priority queue, or perform a lazy cleanup during each new event processing.
Follow-up questions
- How would you handle the case where multiple events of the same type arrive for the same ID (e.g., duplicate price updates)?
- What changes would be necessary if the system needed to scale across multiple instances (distributed state)?
- How would you measure the performance of this system, specifically regarding the latency between the arrival of the second event and the output?
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.