Design a Website Monitoring Service

Scenario

Design a multi-region synthetic monitoring platform (similar to Pingdom or Datadog Synthetics) that allows users to monitor the availability, response time, and correctness of their public websites and APIs.

Requirements

Functional Requirements

• Check Configuration: Users can register endpoints with custom HTTP methods, headers, payload validation rules (e.g., status code 200, response body regex), check intervals (10s to 10m), and selected geographic probe regions.
• Distributed Probing: Lightweight probe workers in multiple global regions execute checks against configured endpoints according to their schedule.
• Alerting: Alert users via webhooks/email/SMS when an endpoint fails. Alerts must avoid triggering on localized, transient network blips.
• Dashboard & Reporting: Provide real-time and historical uptime percentages, latency percentiles (p50, p95, p99), and failure logs.

Non-Functional Requirements

• Low False-Positive Rate: Transits, BGP flaps, or single-probe issues must not generate false alarms.
• High Availability & Scalability: The scheduler and ingestion pipeline must handle checks continuously without missing probe windows.
• Alert Latency: Notifications must be dispatched within 30 seconds of a confirmed failure condition.

Scale & Estimations

• Monitored Endpoints: 5,000,000 targets.
• Average Check Interval: 60 seconds, resulting in ~83,000 executions per second globally.
• Regions: 8 distinct cloud/edge regions running active probes.
• Data Retention: Raw check logs retained for 14 days; downsampled hourly/daily metrics retained for 1 year.

Key Areas to Address

1. Scheduler & Execution Architecture: How checks are assigned, scheduled, and dispatched to probe nodes across multiple regions without thundering herd problems.
2. False-Positive Mitigation: How the system confirms a failure (e.g., cross-region quorum, local retries) before firing an alert.
3. Data Ingestion & Storage: Schema and storage strategy for high-throughput time-series results and raw execution logs.
4. Alerting Pipeline: Stateful alert rule evaluation, deduplication, and suppression logic.