Katkat
Back

Design a concurrent latency percentile tracker

NetflixSoftware Engineer · Technical Screen
HardSoftware EngineerTechnical ScreenSoftware Engineering Fundamentals

Last updated

This problem tests your ability to design a high-throughput, thread-safe data structure capable of tracking request latencies and calculating arbitrary percentiles (e.g., p99) within a sliding time window.

Problem Statement

Design a class LatencyTracker that records the latency of service requests and provides the ability to query the p99 latency over the last 60 seconds. The system must be highly concurrent, as it will be called by thousands of threads simultaneously.

Implement the following interface:

class LatencyTracker:
    def __init__(self, window_size_seconds: int):
        """Initialize the tracker with a sliding window size."""

    def record_latency(self, latency_ms: int):
        """Log a latency measurement in milliseconds."""

    def get_p99(self) -> float:
        """Return the 99th percentile latency over the last window_size_seconds."""

Constraints

  • record_latency should be non-blocking or use fine-grained locking to minimize contention.
  • The system must handle high write throughput (millions of calls per second).
  • Memory usage should remain bounded regardless of the number of requests.
  • Precision vs. performance trade-offs are expected; approximate results are acceptable if documented.

Example

If window_size_seconds is 60, record_latency is called 100 times with values 1 to 100, get_p99() should return approximately 99.0.

Considerations

  • How will you handle data expiration as time progresses?
  • How do you balance the overhead of locking with the accuracy of the percentile calculation?
  • Can you use existing concurrent primitives (like AtomicLong or ConcurrentSkipListMap) to simplify the implementation?

Hints

Hint 1

Consider bucketizing latency values into a fixed-size histogram to represent the distribution rather than storing every raw latency value.

Hint 2

Divide the sliding window into smaller discrete intervals (e.g., 1-second slices) and maintain a thread-safe count for each latency bucket within those slices.

Hint 3

To achieve high concurrency, use an array of atomic counters for each bucket/interval pair to avoid heavy global locks.

Follow-up questions

  • How would your design change if you needed to support p99.9 or p99.99 percentiles instead of just p99?
  • What is the impact of a 'Stop-the-world' garbage collection event on the accuracy of your latency measurements?
  • If you were to distribute this tracker across multiple nodes, how would you aggregate the results to provide a global percentile view?