Katkat
Back

Triage a Suddenly Slow Production Service

ByteDanceSite Reliability Engineer · Technical Screen
EasySite Reliability EngineerTechnical ScreenSoftware Engineering Fundamentals

Last updated

This question tests your structured approach to incident response and root cause analysis in a distributed production environment.

Scenario

You are an SRE on call. Your monitoring dashboard alerts you that the P99 latency for the 'User-Profile-Service' has spiked from 50ms to 800ms over the last 10 minutes. This service is a critical path dependency for the homepage.

Task

Walk me through your step-by-step triage process to identify the root cause and restore service. You should address the following:

  1. Scoping: How do you determine if this is a global outage or isolated to a specific region, data center, or user segment?
  2. Analysis: What specific metrics, logs, or traces would you inspect to differentiate between internal bottlenecks (e.g., CPU/Memory, thread contention) and external dependencies (e.g., database latency, downstream API timeouts)?
  3. Mitigation: Once you have a hypothesis, what strategies would you use to restore service quickly? Discuss the order of operations between mitigation (e.g., traffic shedding, rollback, scaling) and evidence preservation.
  4. Recovery: What metrics or signals must return to healthy states before you declare the incident resolved?

Constraints

  • Assume the service is deployed across multiple Kubernetes clusters.
  • You have access to distributed tracing (Jaeger/Zipkin), structured logs (ELK/Splunk), and infrastructure metrics (Prometheus/Grafana).
  • Prioritize the 'Time to Restore' while ensuring you do not destroy forensic evidence needed for the post-mortem.

Hints

Hint 1

Start by establishing the 'blast radius' to decide if you need to escalate or perform emergency traffic diversion.

Hint 2

Consider the 'golden signals': latency, traffic, errors, and saturation, and how comparing them across different service versions or zones can isolate the issue.

Hint 3

Always differentiate between reactive mitigation (restoring service) and proactive root cause analysis (finding the 'why').

Follow-up questions

  • How would you handle a situation where the service latency is high, but the error rate remains at zero?
  • If a recent deployment is suspected as the cause, what are the trade-offs between a 'rollback' and a 'hotfix'?
  • How do you design your service or observability stack to make this type of triage easier in the future?