How You Build Service Stability and Troubleshoot Production Incidents
ByteDanceSite Reliability Engineer · Technical ScreenLast updated
Tests an SRE candidate's ability to design proactively stable services and execute structured, mitigation-first production incident management under high-throughput conditions.
Scenario
You are an SRE responsible for a critical backend tier in a high-scale feed ecosystem (processing 500,000 queries per second at peak across multiple global regions). The service sits between the API gateway and multiple heterogeneous downstream dependencies, including machine learning ranking models, distributed key-value caches, and user graph services.
Describe your end-to-end framework across two interconnected phases:
- Building & Maintaining Long-term Stability (Pre-incident)
- Triage, Mitigation, and Post-Incident Follow-through (Live incident & Post-incident)
Section 1: Architectural & Operational Stability
Walk through the engineering practices and mechanisms you implement to maintain high availability:
- Service Level Objectives (SLOs) & Error Budgets: How do you define meaningful SLIs (availability, latency) for a multi-tenant tiered service, and how do error budget burn rates gate continuous delivery pipelines?
- Resilience Engineering: Explain concrete implementations of load shedding, graceful degradation, circuit breakers, and bounded concurrency / bulkheading to protect your tier from downstream brownouts or sudden upstream traffic surges.
- Safe Deployments & Change Management: Detail how canary deployments, automated rollbacks, traffic mirroring, and dynamic feature flagging are structured to catch regressions before full blast radius impact.
- Observability Strategy: What specific metrics (e.g., RED/USE methods), distributed tracing contexts, and structured logs are non-negotiable for rapid diagnostics at this scale?
Section 2: Production Incident Troubleshooting
Assume an alert fires at peak traffic: "P99 latency has spiked from 45ms to 850ms, and HTTP 5xx error rate has breached 3% globally."
Walk through your step-by-step troubleshooting methodology:
- Triage & Blast Radius Containment: What are the first three checks you run within the first 5 minutes? How do you prioritize user-facing mitigation over finding the exact root cause?
- Mitigation Levers: What emergency actions (e.g., rolling back recent canaries, flipping feature flags to disable heavy ranking features, shedding non-essential load, rerouting global DNS/edge traffic) do you evaluate?
- Systematic Isolation: If recent deployments and config changes are ruled out, how do you systematically isolate whether the bottleneck is saturated host resources (CPU, memory, epoll/connection pools), an unresponsive downstream dependency, or asymmetric data skew (hot partition / popular creator)?
- Postmortem & Prevention: What does your post-incident process look like to ensure action items are prioritized, blameless reviews are maintained, and systemic failure modes are designed away?
Hints
Hint 1
Always articulate a clear priority order during active outages: stabilize the system and restore user experience first, debug the underlying root cause second.
Hint 2
Structure pre-incident stability around the four golden signals (latency, traffic, errors, saturation) and tie deployment gates directly to fast-burn error budget alerts.
Hint 3
When isolating latency spikes, divide the problem domain quickly by contrasting edge/ingress metrics against downstream dependency durations and local runtime bottlenecks (e.g., garbage collection or thread pool exhaustion).
Follow-up questions
- If your downstream dependencies are healthy but host CPU utilization is normal while request latency continues to climb, what kernel or networking metrics would you inspect next?
- How would you handle an incident where a faulty deployment corrupted persistent state in the cache or database, rendering a simple binary rollback insufficient?
- How do you resolve organizational conflict when product teams resist freezing deployments despite an exhausted error budget?
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.