Katkat
Back

Troubleshoot a Failing Kubernetes Pod: Where to Start and What to Check

AppleSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSoftware Engineering Fundamentals

Last updated

Explain how to troubleshoot a Kubernetes pod that has a problem, starting from its status, container states and events. Tests mapping Pending, ImagePullBackOff, CrashLoopBackOff, OOMKilled and not-Ready states to likely causes, using logs, debug containers and node checks, and separating mitigation from root cause.

A pod in a Kubernetes cluster has a problem. Where do you start investigating, and how do you work from the first thing you see to the root cause?

No symptom is given, so finding out what is actually wrong is part of the task. Explain the checks you run, what each one can tell you, and how your next step depends on what you find.

Constraints and Clarifications

  • Assume kubectl access to the pod's namespace. Ask before relying on shell access to nodes, a metrics system or a log aggregator.
  • Assume the pod is managed by a controller such as a Deployment, so a deleted pod is replaced.

Clarifying Questions

  • What does the problem look like: the pod never starts, it keeps restarting, it runs but never becomes Ready, or it is Ready but its clients see errors or slow responses?
  • Is one pod affected, every replica of the workload, or every pod on one node?
  • Did it begin after a deployment, a configuration or Secret change, or a change to the cluster?
  • Is this serving production traffic, where restoring service comes before finding the cause?

Hints

Hint 1

Let the pod's status choose the path: Before reading any application logs, look at what Kubernetes itself reports about the pod: its phase, its containers' states and the recent events. Different states point to different layers.

Hint 2

The cause may sit outside the container: A pod can fail because of where it was scheduled, the image it pulls, the configuration it references, its resource limits, its probes or the network around it, not only because of its code.

Follow-up questions

  • The pod is in CrashLoopBackOff and its logs are empty. What next?
  • The readiness probe fails only under load. How do you tell a slow application from a badly configured probe?
  • The pod is Running and Ready, but clients calling it through its Service time out. Where do you look?
  • Only pods on one node fail. How do you confirm a node problem and move the workloads off it safely?