Katkat

Design a multithreaded event logger

DatabricksSoftware Engineer · Onsite

Last updated

This problem tests your ability to design a thread-safe, high-throughput logging system that balances memory efficiency with I/O performance under concurrent load.

Problem Statement

Design a thread-safe EventLogger class that allows multiple threads to log messages simultaneously. The logger must ensure that logs are written to an underlying Sink (an interface with a write(message: String) method) in a way that minimizes contention while maintaining chronological ordering based on the time the logging call was made.

Requirements

  1. Thread Safety: Multiple threads must be able to call log(message) concurrently without data corruption.
  2. Ordering: Messages should ideally appear in the Sink in the order they were submitted.
  3. Performance: Minimize the time a thread spends blocked waiting for I/O. Consider using an asynchronous or buffered approach.
  4. Durability: Ensure that even if the application shuts down, pending logs in the buffer are flushed to the Sink.

Interface

class EventLogger:
    def __init__(self, sink: Sink, buffer_size: int):
        pass

    def log(self, message: str):
        """Adds a message to the logger."""
        pass

    def shutdown(self):
        """Flushes remaining logs and stops the logger."""
        pass

Constraints

  • sink.write is a blocking I/O operation.
  • The system must handle high throughput without overwhelming memory if sink.write becomes slow.

Examples

  • Input: Thread A calls log("A"), Thread B calls log("B") shortly after.
  • Output: Sink receives "A" then "B".

Edge Cases

  • The logger buffer is full: should it block the caller or drop logs?
  • Sudden application termination before shutdown is called.

Hints

Hint 1

Consider using a producer-consumer pattern with a thread-safe blocking queue to decouple logging threads from the I/O-bound sink.

Hint 2

Use a dedicated background thread to drain the queue and write messages to the sink sequentially.

Hint 3

Implement the shutdown method to signal the background thread to flush remaining items before exiting.

Follow-up questions

  • How would you modify the design if you needed to support log levels (INFO, WARN, ERROR) and filter them based on a dynamic configuration?
  • What would change if the system requires persistence, meaning logs must be written to disk before the log method returns?
  • How would you handle a scenario where the sink becomes unresponsive for a long period?