Katkat
Back

Design an hourly weather crawling service

HubSpotSoftware Engineer · Onsite
EasySoftware EngineerOnsiteSystem Design

Last updated

This question evaluates a candidate's ability to design scalable, reliable data ingestion and delivery systems, covering competencies such as scheduling and crawling, data normalization, storage schema design, idempotent ingestion, rate limiting, retries, and monitoring.

Scenario

Design a service that crawls the U.S. National Weather Service (or a similar public provider) and provides hourly weather data to internal consumers (and optionally an external API).

Requirements

Functional

  1. Fetch weather data hourly for all supported locations (e.g., stations or grid points).
  2. Normalize provider responses into an internal schema.
  3. Store historical data for querying and analytics.
  4. Expose data to clients via an API (or data export) with predictable freshness.

Non-functional

  • Respect provider constraints (rate limits, robots/terms).
  • High reliability with retries/backoff.
  • Idempotent ingestion (avoid duplicates).
  • Monitoring and alerting on missing/late data.

Assumptions (you may refine)

  • 10,000 locations.
  • Each location updates once per hour.
  • Provider endpoints may be flaky and occasionally return partial data.

Deliverables

  1. Architecture for scheduling and crawling
  2. Storage and schema
  3. Freshness guarantees and backfill strategy
  4. Rate limiting, retries, and idempotency
  5. Monitoring, alerting, and data quality checks