Katkat
Back

Design a Telemetry Pipeline That Handles Inconsistent Metric Names from Old Clients

AnthropicSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

Design a telemetry pipeline that ingests metrics from a large fleet of deployed clients, then handle legacy clients that cannot be updated and report the same metric under inconsistent names. It tests durable ingestion, stream processing and storage design, canonical metric catalogs, version-scoped name mapping and schema evolution.

Design a telemetry system for a data infrastructure team. A large fleet of deployed client software reports telemetry: each record names what is being measured in a telemetry_name field (for example a request latency, an error count or a feature usage event) and carries a value, a timestamp and some context about the client. Engineers need to query this data for dashboards, investigations and alerts.

The interviewer then added a follow-up: many older clients are already deployed in the field and cannot be updated, and they report the same measurements under inconsistent names. The same latency metric might arrive under several spellings and naming conventions depending on the client version. How does your system handle this, and how does it handle schema evolution in general?

Clarifying Questions

  • What kinds of clients are these (desktop or mobile apps, services, devices), and roughly how many are active?
  • How many records per second arrive at peak, and how large is each record?
  • How fresh must the data be for dashboards and for alerts?
  • How long must raw data be retained, and is sampling acceptable for high-volume metrics?
  • Do records contain anything sensitive that must be filtered out or restricted?
  • Do the inconsistent names differ only in spelling and convention, or can the same name mean different things (or use different units) in different client versions?

Part 1 — The core telemetry pipeline

Design the path from clients to queryable storage: how clients send data, ingestion, buffering, processing, storage, and the query and alerting layers. Give rough capacity estimates using stated assumptions.

What This Part Should Cover

  • Client-side batching, retries and backpressure, and the ingestion API
  • A durable buffer that decouples ingestion from processing and allows replay
  • Storage choices for raw records compared with aggregated, queryable data, and how each is partitioned
  • Capacity estimates and the dashboard and alerting read paths

Part 2 — Legacy names and schema evolution

Handle the inconsistently named telemetry from clients you cannot change, without breaking existing dashboards, and design how names and schemas evolve from now on so the problem does not recur.

What This Part Should Cover

  • A canonical metric catalog, and a mapping from raw names to canonical names that is maintained as data
  • The choice between normalizing at ingestion time and at query time, and the effect on history
  • Detecting new unknown variants, and handling names whose meaning or unit differs by version
  • Governance for new clients: registration, versioning and deprecation

Hints

Hint 1

Separate collection from use: Consider which parts of the system must never lose or block on incoming data, and which parts can be rebuilt later from a durable copy of what arrived.

Hint 2

Where should the fix live: You cannot change the old clients, so decide at which stage between arrival and query a raw name becomes a canonical one, and what information is needed to make that decision correctly.

Follow-up questions

  • A newly deployed client version starts sending ten times more records than expected. How does the system protect itself?
  • How would you deduplicate records that a client retried after a timeout?
  • A mapping rule was wrong for two weeks. How do you correct the historical data and the dashboards built on it?
  • How would you let teams add new telemetry without a central team becoming a bottleneck?