Katkat
Back

Design Telemetry Ingestion and Reconcile Metric Names

AnthropicSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSystem Design

Last updated

Design a scalable telemetry ingestion pipeline that standardizes and reconciles metric naming inconsistencies and schema drift across disparate client libraries, historical stores, and real-time query engines.

Scenario

Your company operates a global AI developer platform with client SDKs deployed across multiple languages (Python, TypeScript, Go, Rust) and internal microservices running inference and evaluation workloads. Over several years of rapid development, telemetry metrics have suffered from naming divergence, schema drift, and unit inconsistencies (e.g., req_latency_ms, request_duration_seconds, api.latency).

You need to design a metric ingestion and storage architecture that reconciles metric names and units across newly ingested telemetry, historical datasets, and query consumers (dashboards, alert rules, billing analyzers) without breaking backward compatibility or losing metric fidelity.

Requirements

Functional Requirements

  • Metric Ingestion: Ingest raw metrics (gauges, counters, histograms) tagged with labels from external SDKs and internal services.
  • Schema Reconciliation & Aliasing: Map legacy, variant, or deprecated metric names and units to canonical metric definitions based on configurable rules (e.g., mapping req_latency_ms / 1000 to canonical api_request_duration_seconds).
  • Unified Querying: Allow operators and customers to query using either canonical names or legacy aliases and receive consistent, continuous time-series across both old and new data.
  • Schema Registry & Evolution: Provide an administrative interface or API to define, test, and version metric mapping rules without requiring client updates.

Non-Functional Requirements

  • Throughput: Handle 1,000,000 metric points per second at peak.
  • Latency: End-to-end ingestion latency under 5 seconds from client emission to query availability. Query P95 latency under 500 ms for 24-hour windows.
  • Correctness & Auditability: Preserve original, raw telemetry alongside canonicalized data or maintain a deterministic, reversible lineage for audit and debugging.
  • Storage Retention: 30 days of high-resolution data; 1 year of downsampled rollups.

Deliverables

  1. High-Level Architecture: End-to-end pipeline covering ingestion, stream processing/normalization, time-series storage, and query resolution.
  2. Normalization Strategy: Trade-offs between write-time normalization, query-time rewriting, and materialized view/read-time virtual schemas.
  3. Data Model & Schema Registry: Schema representation for metrics, tags, canonical mappings, and unit transformation expressions.
  4. Failure Modes & Edge Cases: Handling unknown/unmapped metrics, metric collisions (two distinct metrics mapping to one name), unit mismatches, and rule deployment rollbacks.

Hints

Hint 1

Consider the trade-offs between write-time transformation (lower query cost, storage overhead, permanent schema choices) versus read-time query expansion (flexible mapping changes, higher query latency).

Hint 2

Design a schema registry with local caching in the stream-processing tier to avoid hitting a centralized metadata database on every ingested event.

Hint 3

Think about how to handle unit conversions (e.g., milliseconds to seconds) alongside simple name aliasing so that downsampled aggregates remain mathematically valid.

Follow-up questions

  • How would you handle a collision where two separate teams emitted `token_count` with different semantic definitions (input tokens vs total tokens)?
  • If an operator adds a new canonical mapping today, how would you make 6 months of historical data queryable under this new name without running a full database backfill?
  • How can you ensure that streaming aggregators (e.g., tumbling windows for rate/sum calculations) don't compute incorrect results during the rollout of a unit conversion rule?