Katkat
Back

Design Monitoring and Reliability for 99.99% Availability

AtlassianSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

Design a comprehensive observability and reliability framework to guarantee 99.99% availability for a multi-tenant cloud collaboration platform. The design must cover Service Level Indicators (SLIs), Service Level Objectives (SLOs), automated burn-rate alerting, dependency fault isolation, and chaos recovery testing.

Scenario

You are an engineer at a cloud collaboration platform (such as Jira or Confluence). Several tier-0 enterprise microservices (e.g., Issue Tracking, Permissions, and Real-time Search) must meet a strict Service Level Agreement (SLA) of 99.99% availability ("four nines") over rolling 30-day windows. At four nines, your service is permitted no more than approximately 4.32 minutes of downtime or degraded performance per month.

Design the monitoring architecture and reliability mechanisms required to sustain and verify this target under continuous production traffic.

Requirements

Functional

  • Service Level Framework: Define quantitative SLIs and SLOs grounded in the Four Golden Signals (Latency, Traffic, Errors, Saturation).
  • Error Budget & Burn-Rate Alerting: Establish an alerting framework that triggers before an outage consumes the monthly error budget, avoiding alert fatigue while minimizing Time to Detect (TTD).
  • Dependency & Blast Radius Control: Design mechanisms to isolate failures in downstream dependencies (third-party APIs, shared relational databases, caching tiers) so that degradation does not breach the tier-0 SLO.
  • Automated Mitigation: Automatically remediate standard failure modes (e.g., shedding non-critical load, circuit breaking, rerouting traffic).
  • Testing & Verification: Formulate an ongoing validation strategy (such as automated chaos experiments or canary deployments) to verify that recovery time objectives (RTO) remain under target.

Non-Functional & Scale

  • Peak Traffic: 100,000 incoming requests per second (RPS) distributed across 3 global regions.
  • Observability Latency: Metrics must be collected, aggregated, and queryable by alert evaluators within 15 seconds of emission.
  • Independence: The monitoring and mitigation system must have minimal shared failure domains with the services it monitors.

Key Design Areas to Cover

  1. Metric Collection & Pipeline: Architecture of metric agents, streaming aggregation, timeseries storage, and evaluation engines.
  2. SLI/SLO Math & Alerting Rules: How to calculate error budgets and configure multi-window, multi-burn-rate alerts (e.g., 2% budget consumed in 1 hour vs. 5% consumed in 6 hours).
  3. Reliability Architecture: Circuit breakers, adaptive rate limiting, bulkhead patterns, and cell-based architecture to constrain blast radius.
  4. Recovery & Chaos Framework: Automated rollback mechanisms and continuous chaos testing in staging/production.

Hints

Hint 1

Four nines means high sensitivity to short incident bursts; simple static threshold alerts on raw error counts either page too frequently or react far too late.

Hint 2

Decouple your monitoring and control plane from the core product path so that an outage in the application platform does not disable telemetry or auto-remediation.

Hint 3

Structure downstream dependencies into hard and soft dependencies, using graceful degradation (e.g., serving cached/stale data or disabling expensive widgets) to preserve the core user journey.

Follow-up questions

  • How do you resolve conflicts between CAP theorem guarantees (consistency vs. availability) when your primary database region becomes partitioned?
  • How would you instrument Synthetic Monitoring (probers) versus Real User Monitoring (RUM) to prevent client-side network issues from polluting your internal SLOs?
  • If an automated remediation script (e.g., auto-scaling or pod restarts) triggers a cascading thundering herd, how should the system detect and halt the remediation loop?