Katkat
Back

Use Metrics and Logs to Establish a Production Root Cause

TeslaSoftware Engineer · HR Screen
MediumSoftware EngineerHR ScreenSoftware Engineering Fundamentals

Last updated

This question tests your systematic approach to troubleshooting production incidents in distributed architectures using metrics, distributed tracing, structured logs, and change audit trails.

Scenario

You are on call for a high-throughput telemetry ingestion platform responsible for streaming sensor events from thousands of connected devices. The architecture consists of an API Gateway (Go), a message broker, a set of stream processing workers, and an Elasticsearch cluster backed by a relational metadata store.

At 14:15 UTC, the on-call engineer receives an automated paging alert: the API Gateway p99 response latency has spiked from 40ms to over 6,000ms, and downstream consumers are reporting a high rate of 504 Gateway Timeout errors. Simultaneously, alerts trigger for elevated CPU utilization across the worker pool and an accumulation of pending connection pools.

Task

Explain how you would systematically triage this incident, isolate the true root cause, and differentiate it from cascading secondary symptoms.

In your response, address the following:

  1. Immediate Triage and Containment: What are the first telemetry signals you check to assess customer impact, and what immediate actions might mitigate impact before finding the full root cause?
  2. Correlating Observability Signals: How do you utilize RED metrics (Rate, Errors, Duration), distributed trace IDs, and structured log aggregation to pinpoint where the bottleneck originated?
  3. Differentiating Cause from Symptom: High CPU, thread pool exhaustion, and connection timeouts are often symptoms rather than root causes. Provide a concrete example of how you verify whether a database lock, a faulty deployment, or an upstream traffic surge caused the system failure.
  4. Verification and Post-Mortem Evidence: What quantifiable evidence is required to confirm the incident is resolved, and what systemic safeguards would prevent recurrence?

Hints

Hint 1

Begin by cross-referencing the incident onset timestamp with the change history, including code deployments, configuration flags, and infrastructure scaling events.

Hint 2

Follow a single trace ID across the call graph to isolate the specific hop where duration diverges from the baseline.

Hint 3

Distinguish upstream volume changes from downstream latency degradation by checking whether request rates spiked prior to latency increases or if throughput dropped as a result of saturation.

Follow-up questions

  • How do you trace request context across asynchronous message boundaries (e.g., Kafka topics) where standard synchronous HTTP headers are not automatically forwarded?
  • If your log aggregation pipeline (e.g., Elasticsearch) itself becomes saturated or unresponsive during an incident, what fallback mechanisms do you rely on?
  • How would you implement automated canary analysis or circuit breakers to detect and isolate this failure pattern before it degrades the entire cluster?