Katkat
Back

Finding the Root Cause of an Intermittent Failure When the Obvious Metrics Look Healthy

AmazonSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteBehavioral & Leadership

Last updated

A behavioral debugging question: describe a system that failed intermittently while its obvious metrics looked healthy, how you found the root cause, and how long it took. It tests whether you can explain why the dashboards stayed green and show a systematic, evidence-driven narrowing process.

Tell me about a time when a system was failing intermittently, and the very obvious metrics were all looking good. How did you find the root cause, and how long did it take?

This was one of three behavioral questions in an onsite round. When the first answer did not satisfy the interviewer, they restated the same question in different words and asked it again. Expect to be pushed until the debugging path itself is concrete: what you looked at, in what order, and what finally exposed the cause.

Clarifying Questions

  • Does it have to be a system I owned, or can it be a dependency I debugged on behalf of my team?
  • Should the time I give include detecting the problem, or only finding the root cause once it was noticed?
  • How much technical detail do you want about the root cause itself?

Hints

Hint 1

Explain why the dashboards stayed green: A strong story names the specific reason the obvious metrics missed the failure. Think about what the standard dashboards measure, where they measure it, and what they average away.

Hint 2

Show the narrowing: Describe how you went from "something fails sometimes" to one component, one condition and one mechanism, including at least one hypothesis that turned out to be wrong.

Follow-up questions

  • Which signal finally exposed the failure, and why was no alarm watching it?
  • What was your first hypothesis, and what evidence ruled it out?
  • What made the investigation take as long as it did, and what would have shortened it?
  • What did you change afterward so that this class of failure appears on a dashboard or an alarm?