On-Call Case Study: Debugging a Misbehaving Production Server

You are the on-call engineer for a backend service, and you have just been paged about one of its production servers. The interviewer plays the role of the system and of your teammates: you learn what the dashboards, logs and commands show only by asking for them, so the quality of your questions drives the exercise.

The specific incident used in the interview was not described. For practice, assume the page says that one server in a fleet behind a load balancer is returning slow and failed responses, and that nobody yet knows why. Talk through your investigation out loud.

Constraints and Clarifications

• Treat it as a live incident: user impact matters as much as finding the root cause.
• You may ask for any metric, log line or command output, but ask for specific things rather than "everything".
• You may mitigate before you fully understand the cause, but say what each action costs.

Clarifying Questions

• Is it only this server, or are other servers in the same fleet or region showing the same symptoms?
• When did it start, and was there a deploy, configuration change or traffic change around that time?
• What exactly does the alert measure (error rate, latency, a health check), and what are the normal values?
• What does the service depend on (databases, caches, other services), and are those healthy?

Part 1 — Triage and impact

What do you do in the first few minutes after the page? Explain what you check first, how you decide how many users are affected, who you tell, and whether to mitigate before you understand the cause.

What This Part Should Cover

• Acknowledging the page and establishing scope: one host, the whole fleet, or a dependency
• Sizing user impact, and declaring an incident or escalating when warranted
• Checking recent changes as prime suspects
• A fast, low-risk mitigation option and its cost

Part 2 — Find the cause

Narrow the problem down on the server and in the service. Which signals and commands would you ask for, in what order, and how would you turn the answers into hypotheses you can confirm or rule out?

What This Part Should Cover

• A systematic pass over host resources: CPU, memory, disk space and I/O, network, file descriptors and process state
• Application signals: error logs, thread or connection pools, garbage collection, dependency latency
• Explicit hypotheses, each tested with a specific observation
• Comparison of the sick host with a healthy peer

Part 3 — Mitigate, fix and follow through

Once you have a likely cause, how do you restore service, confirm the fix worked, and make sure the problem does not come back?

What This Part Should Cover

• Choosing between draining the host, restarting, rolling back or fixing forward, with trade-offs
• Capturing evidence before destructive actions
• Verifying recovery with the same signals that fired the alert
• Communication during the incident and a blameless follow-up with concrete action items