Katkat
Back

Design Camera Health Monitoring

VerkadaSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSystem Design

Last updated

This question evaluates a candidate's competency in designing scalable, near-real-time monitoring systems, covering high-throughput heartbeat ingestion, compact per-device state management, detection of missing events, and handling duplicates, late arrivals, and ordering issues.

Design a system that monitors the health of 10 million cameras and continuously reports, in near real time, how many devices are healthy and how many are unhealthy.

Each camera sends one heartbeat every minute. A camera is considered healthy if the system has received at least one heartbeat from it within the last 2 minutes; otherwise it is unhealthy. The dashboard needs a live global count (and, where it helps, breakdowns), not per-camera time-series history.

Your design should address:

  • The heartbeat ingestion path (how heartbeats are received, validated, and buffered).
  • Storage for device state (what you keep per camera and where).
  • How to compute healthy and unhealthy counts in near real time without rescanning all 10M devices on every tick.
  • How to handle duplicate, late, and out-of-order heartbeats.
  • Scalability, fault tolerance, and monitoring.

Constraints & Assumptions

  • Population: ~10,000,000 provisioned cameras, tracked in a registry, so total_active is well-defined and unhealthy can be derived/cross-checked as total_active − healthy.
  • Cadence: 1 heartbeat/camera/minute ⇒ ~167K heartbeats/sec steady state. Provision for reconnect storms (post-deploy/network-blip retries) at roughly 3–5× steady state.
  • Health window: received a heartbeat within the last 120 s.
  • Freshness: an ops dashboard tolerates a few seconds (≈5–10 s) of count staleness; transactional exactness is not required.
  • Payload: a heartbeat is tiny (camera_id + timestamp + a few optional fields), ~50–100 B of useful data; the scaling pressure is request/connection rate, not bandwidth.
  • Hot keys: one camera = one heartbeat/min, so no single device can be a hot key (relevant to partitioning).
  • Scope: aggregate health counts only — not video ingestion, per-camera dashboards, or per-device alerting (note where the design extends if asked).

Clarifying Questions to Ask

  • How fresh must the counts be — sub-second, or is 5–10 s of lag acceptable? (Drives stream-aggregate vs. periodic rollup, and the consistency model.)
  • Do we need breakdowns by region / firmware / tenant, or only one global number? (Drives counter cardinality.)
  • Is the device population fixed and registered, or can unknown cameras appear? (Determines whether unhealthy = total_active − healthy is valid.)
  • What does "received a heartbeat" mean — server receive time, or the client-stamped time? (Client clocks are unreliable.)
  • What delivery guarantees does the transport give (at-least-once, ordering), and how large can a reconnect storm get (fraction of fleet, over what window)?
  • Are there per-tenant isolation, retention, or compliance requirements on the heartbeat data?

Hints

Hint 1

Where to start: Do the back-of-the-envelope math first. $10^7$ cameras $/\,60\text{ s} \approx 167\text{K}$ heartbeats/sec, but one record per camera is only ~100 bytes, so the *entire* device state is roughly **1 GB** — small. The hard part is not storing state, it's avoiding a per-minute full scan of 10M rows just to recompute two numbers. Separate "ingest is high-throughput" from "state is tiny."

Hint 2

Counting strategy: Re-counting two numbers from 10M rows every tick is the obvious approach — what does it cost, and how stale are the numbers between scans? Now ask the opposite question: in a typical minute, how *many* cameras actually change health state? If that number is tiny compared to the fleet, what would you have to react to (instead of re-scan) to keep the counts current?

Hint 3

The timeout problem: A heartbeat arriving is an event you can react to. But "this camera went 2 minutes with *no* heartbeat" is the *absence* of an event — nothing shows up to tell you. How do you detect something that is defined by nothing happening, for 10M cameras, without periodically sweeping all of them? What would you have to set up at the moment a heartbeat *does* arrive?

Hint 4

Idempotency under retries: Retries mean the same heartbeat can arrive twice, late, or out of order — and you may have already scheduled a "this camera expired" check that a newer heartbeat has since made wrong. Which clock do you trust when the client's is unreliable? And when a possibly-stale expiry check eventually runs, what could you store per camera at heartbeat time that lets the check decide for itself whether it's still relevant?

Follow-up questions

  • Extend this to report healthy/unhealthy counts **broken down by region, tenant, or firmware version** without exploding counter cardinality or the read path. What changes in your counters, rollup, and read path?
  • After a consumer crash and partition rebalance, how exactly does the new owner reconstruct in-memory state, the timer schedule, and the counters — and how do you prevent double-counting a transition during replay?
  • How do you distinguish "the cameras are actually unhealthy" (a real outage) from "our monitoring pipeline is unhealthy" (consumer lag making counts stale)? What signals and alerts separate the two?
  • The health threshold is "2 minutes" and the cadence is "once a minute." How would you make these **configurable per tenant or device class** (e.g. a 5-minute cadence, or a 30-second SLA) without redesigning the timeout machinery?