Katkat
Back

Design an observability logging system for emitting and querying logs

BasetenSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

System design question: build an observability logging system that lets services emit structured logs and lets engineers query them. It covers non-blocking client buffering, ingestion with backpressure and a durable buffer, time-partitioned storage, indexing versus scanning trade-offs, query execution limits, and isolating noisy services.

Design an observability logging system. Services running across a fleet of machines emit log records, and engineers query those records, for example while debugging a production incident. The design must cover both capabilities the interviewer asked for: how logs are emitted and how they are queried.

Constraints and Clarifications

  • Scale, retention, query patterns and freshness requirements were not given. Ask, or state your assumptions with numbers.

Clarifying Questions

  • How many services and hosts emit logs, and what are the average and peak volumes in records or bytes per second?
  • Are records structured (fields such as timestamp, service, level and trace ID, plus key-value attributes) or free text?
  • Which queries must be fast: filtering by service, time range and level, full-text search in messages, lookup of one request or trace ID, or aggregations such as error counts per minute?
  • How soon after emission must a record be queryable, and how long must logs be retained? Is older data queried less often?
  • Under overload, may low-severity records be sampled or dropped, or must every record be kept?
  • Do logs contain sensitive data, and who may query which services' logs?

Part 1 — Emitting logs

Design the path from a log call in application code to durable storage: the client-side API or library, the record format, and the ingestion tier.

What This Part Should Cover

  • The logging API and record schema, including the fields every record carries
  • Buffering and batching in the process or on the host, and the policy when buffers fill
  • The ingestion tier: authentication, validation, per-service limits, and a durable buffer
  • Delivery guarantees, duplicates, and how lost records are accounted for

Part 2 — Querying logs

Design how logs are stored and indexed so that engineers can query them, and the query interface they use.

What This Part Should Cover

  • Data layout: partitioning by time and service, compressed chunks, and storage tiers by age
  • The indexing strategy, and its cost at ingest versus its benefit at query time
  • The query API and its execution: filters, text search, time ordering, pagination, aggregation, parallel fan-out
  • Protection against expensive queries, and visibility of very recent records

Hints

Hint 1

The application should not wait: Think about what happens to a request being served when the logging backend is slow or unavailable, and where log records wait in the meantime.

Hint 2

Every query has a time range: Use that fact to decide how data is laid out, then decide what else is worth indexing and what can be scanned.

Hint 3

Indexes cost money too: Indexing every word of every message makes text search fast but also makes ingestion and storage expensive. Compare that with indexing less and scanning more at query time.

Follow-up questions

  • One service starts logging 100 times its normal volume during an incident. What happens across your system, and how do you contain it?
  • An engineer needs every log record for one request ID across all services over the past 30 days. How does your design answer that quickly?
  • How would you add a live tail that streams matching records within seconds of emission?
  • A user asks for their personal data to be deleted, and it appears in compressed, immutable log chunks. How do you handle it?