Katkat
Back

Debug a slow concurrent network server under load

MongoDBSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSoftware Engineering Fundamentals

Last updated

This question tests your ability to diagnose performance bottlenecks in a multi-threaded network server using I/O multiplexing, focusing on how blocking calls and thread management impact latency.

The Problem

You are investigating a reported latency spike in a high-throughput network server. The server uses an event-loop (based on epoll) to handle incoming connections and a fixed-size worker thread pool to process requests.

Under normal operation, the server handles 5,000 requests per second with < 10ms latency. During peak loads of 8,000 requests per second, the P99 latency jumps to over 2 seconds. The server's CPU usage is hovering at 60%, and the system shows no signs of high disk I/O wait. However, you notice that the worker threads are frequently in a WAITING state, and the queue of pending tasks for the worker threads is growing rapidly.

Your Task

  1. Identify the most likely root causes for the increased P99 latency.
  2. Explain how the interplay between the event-loop and the worker thread pool creates a bottleneck.
  3. Propose a monitoring strategy to pinpoint the specific component causing the delay.

Key Considerations

  • The server uses a non-blocking socket model for the event-loop.
  • Consider the impact of "head-of-line" blocking if a subset of requests performs synchronous file logging or external API calls within the worker thread.
  • Consider how context switching and thread contention might manifest given the thread pool size.

Example Scenario

If a single thread in the pool performs a synchronous DNS lookup that hangs for 500ms, how does that affect the overall throughput of the server, given the event-loop is still accepting new connections at a high rate?

Hints

Hint 1

Analyze the difference between the rate at which the event-loop accepts connections and the rate at which the worker pool completes tasks.

Hint 2

Consider whether the worker threads are performing any operations that might inadvertently block the entire pool, such as synchronous logging or mutex contention.

Hint 3

Evaluate if the fixed-size thread pool configuration is sufficient for the burst in request volume or if the task queue management is becoming a serialized bottleneck.

Follow-up questions

  • How would you redesign the system to prevent a slow, long-running task from consuming a worker thread and starving other requests?
  • What specific metrics (e.g., thread wait time, queue depth, event-loop cycle time) would you add to your observability stack to catch this in the future?
  • If you migrated the architecture to a fully asynchronous model (e.g., using coroutines), what new complexities or potential performance pitfalls would you introduce?