Live troubleshooting: diagnose and mitigate very high CPU on a public-facing service

This is an interactive scenario from a senior reliability engineering loop. You are told that a public-facing service is running at very high CPU usage. The interviewer then steps through the situation with repeated "what would you look at next?" prompts and reveals information only when you ask for it. Walk through how you would confirm and scope the problem, narrow down the cause, mitigate it, and follow up.

One mitigation that comes up is rate limiting customers. Be ready to defend or reject it for a service where every API call carries revenue and customer trust, compared with scaling out or moving work to an asynchronous queue.

Constraints and Clarifications

• You do not get the full picture up front; part of the evaluation is which questions you ask and in what order.
• Assume the usual tools are available: dashboards for service and host metrics, logs, the deployment and configuration history, and the ability to profile a running instance.

Clarifying Questions

• Is CPU high on every instance, or only on some (one region, one build version, one host type)?
• Are customers affected yet: are latency and error rates up, or is it only the CPU metric?
• Did anything change recently: a deployment, a feature flag, configuration, or a dependency?
• Has request volume changed, and is any increase concentrated in a few customers or endpoints?
• Does the service autoscale, and is it stateless?

Part 1 — Confirm and scope

What do you check in the first few minutes, and how do you decide how urgent this is?

What This Part Should Cover

• Validation of the signal and of its customer impact
• Scoping by instance, region, version and time of onset
• A severity call and who needs to know

Part 2 — Narrow the cause

Walk through what you would look at next, and how each answer changes your next step.

What This Part Should Cover

• A branching investigation rather than a fixed checklist
• Use of change history, traffic breakdowns and profiling
• Common causes of CPU saturation and how each one is recognized

Part 3 — Mitigate

Choose a mitigation. Compare rate limiting customers, scaling out, and moving work to an asynchronous queue, for a service where rejected calls cost revenue and trust.

What This Part Should Cover

• Rollback when a change is the cause
• Trade-offs among scaling out, targeted throttling, broad rate limits and asynchronous processing
• Protecting downstream dependencies and avoiding a retry storm

Part 4 — After the incident

What do you do once the service is stable?

What This Part Should Cover

• Root cause confirmation and a blameless review
• Capacity, autoscaling and quota changes
• Detection and alerting improvements