Katkat
Back

Diagnose why a scaled system became slow

AtlassianSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSystem Design

Last updated

Tests systematic incident triage, observability strategies, and root-cause analysis across distributed tiers (caching, connection pools, databases, compute) when a scaled service suffers severe latency degradation.

Scenario

You are an engineer on the core platform team for a collaborative project management service (similar to Jira). Following a recent migration of several large enterprise tenants, overall daily active users increased from 5 million to 10 million, and peak request throughput doubled from 50,000 QPS to 100,000 QPS.

Shortly after this milestone, users began reporting severe slowness. Observability dashboards indicate that the core board aggregation endpoint (GET /api/v1/projects/{projectId}/board) experienced a major latency regression:

  • p50 latency rose from 40ms to 180ms
  • p95 latency jumped from 120ms to 2.1s
  • p99 latency spiked from 250ms to 5.4s
  • Error rate (HTTP 504 Gateway Timeout) climbed to 3.2%

Interestingly, the stateless application tier running on Kubernetes reports average CPU utilization of only 42% and memory utilization of 55%. However, downstream read-replica database connection pools are frequently exhausted, and the shared Redis cluster reports a 400% surge in eviction rates.

System Architecture

  1. Edge / Ingress: Global CDN and reverse proxy routing traffic to regional API gateways.
  2. Application Tier: Stateless microservices in Go and Java running in Kubernetes, handling business logic, permission checks, and aggregation.
  3. Cache Tier: Clustered Redis used for session data, tenant configuration, and hydrated issue summaries.
  4. Data Tier: Relational primary-replica database cluster (PostgreSQL) storing project boards, issue transitions, and permissions, paired with Elasticsearch for complex text searches.

Expectations

Walk through how you would diagnose, triage, and permanently resolve this performance collapse. Your response should cover:

  1. Triage & Incident Mitigation: Immediate steps to stabilize the system and protect end-user experience before identifying the exact code-level cause.
  2. Diagnostic Strategy: The specific metrics (RED/USE methods), distributed tracing spans, and log patterns you would inspect to isolate which tier is the primary bottleneck.
  3. Failure Mode Analysis: A deep-dive into plausible architectural bottlenecks across:
    • Application runtimes (thread starvation, garbage collection, connection pooling)
    • Cache behavior (hot partitions, cache stampedes, serialization overhead, key eviction)
    • Database tier (lock contention, unindexed queries on large tenants, replica lag, IOPS saturation)
  4. Remediation & Long-Term Architecture: Architectural and data-model changes to support the new scale cleanly without recurrent regressions.

Hints

Hint 1

Consider why low CPU usage combined with high latency and timeouts typically signifies thread blocking or wait-states on external resources rather than compute exhaustion.

Hint 2

Examine cache hit ratio and eviction metrics together; high evictions combined with sudden large enterprise tenants usually signal noisy-neighbor skew or oversized payloads bursting memory limits.

Hint 3

Inspect the database connection lifecycle; simply increasing connection pool size often degrades PostgreSQL performance due to process context-switching and lock contention.

Follow-up questions

  • How would you address a cache stampede (thundering herd) scenario when a critical tenant-wide cache key expires during peak traffic?
  • Why does increasing `max_connections` on a PostgreSQL database often degrade throughput under high load, and how does connection pooling (e.g., PgBouncer) mitigate this?
  • What architectural changes (e.g., tenant sharding, read-side aggregation, asynchronous board pre-computation) would you introduce to eliminate noisy-neighbor issues for multi-tenant board views?