Katkat
Back

Optimize Object-Backed APIs and Design an LLM Observability Platform

LangchainSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

Design a high-throughput LLM observability and tracing platform that reliably captures spans, aggregates token billing, and provides low-latency trace retrieval over an object-storage backend.

Scenario

Modern generative AI applications execute multi-step chains, tool invocations, and LLM calls for every user interaction. As traffic scales, observability platforms must ingest deep trace trees, extract token counts for accurate customer billing, and allow developers to inspect full execution graphs without excessive latency or cloud storage egress costs.

You are tasked with designing the core architecture for an LLM observability platform that handles both high-throughput trace ingestion and interactive querying, backed primarily by cloud object storage.

Requirements

Functional

  • Trace Ingestion & Retrieval: Ingest execution spans (prompts, completions, tool parameters, latencies). Allow retrieving specific runs or sub-trees by run_id with sub-second latency.
  • Token Billing & Aggregations: Accurately compute token counts per tenant, model, and project for billing and usage quotas.
  • Real-Time Metrics & Alerting: Track p50/p95/p99 latencies, error spikes, and trigger alerts when cost or failure thresholds are breached.
  • Optimized Storage Access: Prevent full-file scans when querying specific spans or metadata within large object-backed JSON payloads.

Non-Functional

  • Scale: Ingest 50,000 spans per second at peak (~500M spans/day). Average span payload is 20 KB, but some multi-turn/multimodal traces reach up to 5 MB.
  • Latency: Ingestion acknowledgment < 50ms; trace detail retrieval p95 < 200ms; dashboard aggregate queries p95 < 1s.
  • Data Durability & Retention: Traces retained for 90 days. Raw payloads must be cost-effectively stored in object storage (e.g., S3/GCS).

What Your Design Should Cover

  1. API & Ingestion Pipeline: Endpoints for span collection, batching, and decoupling ingestion from writes.
  2. Storage Tiering & Data Model: Schema for search metadata versus raw payload blobs; techniques to avoid scanning large blobs for partial reads (e.g., manifests, chunking, byte-range requests).
  3. Aggregation & Billing Pipeline: Stream processing for deduplication, exact token accounting, and metric rollups.
  4. Trade-offs: Latency vs. storage cost, consistency vs. availability during traffic spikes, and index density.

Hints

Hint 1

Decouple metadata indexing from payload storage so queries can filter by tenant, tag, or latency without touching object storage.

Hint 2

To avoid downloading and parsing multi-megabyte JSON payloads, store span indices with byte offsets or use columnar formats that support HTTP range requests.

Hint 3

Use a stream processor with idempotency keys for token billing to guarantee exact-once semantics even during client retry storms.

Follow-up questions

  • How would you handle deduplication if clients automatically retry timed-out batch trace submissions?
  • If raw trace payloads contain sensitive PII or API keys, where and how would you implement redaction without hurting ingestion throughput?
  • How does your design support running batch evaluation or anomaly detection jobs over 30 days of historical traces without degrading real-time dashboard performance?