Katkat
Back

Design a scalable logging system

RipplingSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

This question evaluates a candidate's ability to design scalable, reliable logging and observability infrastructure, testing competencies in distributed systems, ingestion pipelines, indexing and data modeling, storage and retention policies, multi-tenancy, and performance/reliability trade-offs.

Design a centralized logging system for a large organization — an internal observability platform (think a self-hosted Splunk / Datadog Logs / ELK) that many teams ship their application and infrastructure logs to.

The system should:

  • Collect logs from many applications and hosts written in different languages and running in different environments (VMs, containers, serverless).
  • Support near-real-time search and filtering (e.g. by service, host, time range, and severity), plus free-text search on the message.
  • Provide configurable retention policies (e.g. 7 days hot/fully-searchable, 90 days archived, then deletion).
  • Handle high write throughput with large, bursty spikes (incident storms, retry loops).
  • Be reliable (no or limited data loss) and multi-tenant (different teams are isolated, and one noisy team cannot degrade another).

Walk through requirements, high-level architecture, the data model and indexing strategy, scaling and partitioning, reliability and delivery semantics, multi-tenancy, cost, and the key trade-offs you'd flag.

Constraints & Assumptions

State your scale numbers explicitly — they drive every partitioning and storage decision. A reasonable large-org sizing to defend:

  • ~10,000 hosts, each emitting ~100 events/sec on average → on the order of ~1M events/sec baseline, bursting to ~5–10M events/sec during incidents.
  • Average serialized event size after enrichment ~1 KB → roughly ~1 GB/s (~86 TB/day) raw ingest.
  • Ingest latency target: a few seconds end-to-end from emit to searchable.
  • Query latency target: interactive — sub-second to a few seconds for a filtered search over the hot window.
  • Logs are append-only, immutable, time-ordered events. No per-record updates/deletes; only bulk expiry by retention.
  • Slight reordering and small windows of duplication are acceptable. At-least-once delivery is the realistic target.

Clarifying Questions to Ask

  • Scale & burst profile — How many hosts/services, what average and peak events/sec, and what burst multiplier should the system absorb without dropping data?
  • Search semantics — Is full-text search on the message body a core requirement, or are structured field filters (service/host/level/time) enough? This decides inverted-index vs. columnar for the hot tier.
  • Retention & compliance — What are the hot/warm/archive windows, are they configurable per tenant, and are there regulatory constraints (PII redaction, immutability, region residency)?
  • Tenancy model — How many tenants, how skewed is their volume (a few whales vs. a long tail), and what isolation guarantee is required (hard isolation vs. logical separation with enforced filters)?
  • Consistency expectations — Is at-least-once with occasional duplicates acceptable, or is exactly-once required? Is bounded ingest lag during an outage acceptable as long as nothing is lost?
  • Budget posture — Is this cost-optimized (logs are mostly write-only) or latency-optimized (long searchable window)? This sets how aggressively to tier and sample.

Hints

Hint 1

Where to start: Treat this as a **streaming pipeline**, not a database problem. Sketch the path an event takes — *producer → host agent → collector → durable buffer → processor → store → query* — and reason about which stage must never block the producer versus which stages are allowed to lag.

Hint 2

The keystone component: What lets a *slow* indexing path coexist with a *write path that must not block apps*? A **durable, partitioned, replicated log buffer** (Kafka/Pulsar) in the middle decouples them and gives you replay/backpressure instead of data loss. Almost every reliability and scaling answer flows from putting this in the right place.

Hint 3

Storage is two tiers, not one: Logs are huge and mostly never read. Resist storing everything in one searchable engine. Think about a small, expensive **hot** tier (inverted-index search, short window) versus a cheap **archive** tier (compressed columnar object storage, queried on demand) — and how *retention* becomes a cheap metadata operation rather than per-record deletes.

Hint 4

Multi-tenancy pitfalls: Where does `tenant_id` come from (hint: not the client payload)? How do you stop one tenant's burst from starving others, and how do you lay out indices so you don't either leak across tenants or explode your shard count?

Follow-up questions

  • A single tenant suddenly produces 100× their normal volume during an incident. Trace exactly what happens at each stage of your pipeline — and which other tenants are (or aren't) affected, and why.
  • A team needs to search logs from 60 days ago by free-text on the message. Your hot window is 7 days. How does that query execute, what's the latency, and what could you change to make 60-day free-text search feasible (and at what cost)?
  • Your search cluster goes down for 3 hours during peak traffic. Walk through what's lost, what's delayed, and what the recovery looks like — what sizing decision determines whether you lose data?
  • Compare an inverted-index search engine (Elasticsearch/OpenSearch) against a columnar store (ClickHouse) for the **hot** tier. When would you pick each, and how does the choice change if free-text search is/ isn't a core requirement?
  • How would you prevent a "mapping explosion" from a tenant that logs thousands of distinct high-cardinality JSON keys, without simply refusing their logs?