Katkat
Back

Design a Centralized Logging System

AppleSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

Tests the candidate's ability to design a high-throughput, distributed log collection and search platform, balancing ingestion backpressure, index cost, query latency, and tiered storage.

Scenario

Design a centralized logging platform for a large ecosystem of distributed microservices, edge daemons, and client applications. The system must collect, store, index, and provide fast search access to operational and diagnostic logs generated across millions of host instances.

Requirements

Functional Requirements

  • Log Ingestion: Ingest structured (JSON) and unstructured log messages from client agents across various data centers and platforms.
  • Search & Filtering: Allow engineers to perform fast search queries filtering by timestamp range, service_id, trace_id, log_level, and keyword or regex matches.
  • Live Tailing: Support streaming tailing of logs for a specific service or deployment in near real-time (sub-5 second delay).
  • Retention & Tiering: Store searchable logs for 7 days in "hot" storage, automatically migrating logs to compressed "cold" storage for 90 days before deletion.

Non-Functional Requirements

  • High Write Throughput: Support up to 5 million log events per second at peak.
  • Durability & Fault Tolerance: Ingestion must not lose logs during network partitions or broker outages. Ingestion backpressure must not crash the host service.
  • Query Performance: P95 query latency under 2 seconds for searches targeting the last 24 hours.
  • Cost Efficiency: Optimize indexing and storage overhead, as indexing 100% of text fields at this scale is cost-prohibitive.

Scale & Estimations

  • Peak Ingestion Rate: 5,000,000 logs/sec.
  • Average Log Size: 1 KB uncompressed.
  • Write Bandwidth: ~5 GB/s (40 Gbps) at peak.
  • Daily Volume: ~430 TB/day raw logs; ~3 PB for 7 days of hot storage.

What to Cover

  1. Data Model & API: Schema of a log entry, ingestion API, and search API design.
  2. Ingestion & Buffering: Agent collection strategy (daemon/sidecar), batching, local buffer, and message queuing (e.g., Kafka) to handle traffic spikes.
  3. Indexing & Storage Architecture: Hot vs. cold storage engines, index strategy (e.g., inverted index vs. columnar log storage with metadata labels), and log deduplication/enrichment.
  4. Query & Serving Layer: Query routing, fan-out, scatter-gather execution, and live-tail subscriptions.
  5. Trade-offs: Full-text indexing vs. label-only indexing (e.g., Elasticsearch vs. Grafana Loki), durability vs. ingestion latency, and compression techniques.

Hints

Hint 1

Consider decoupling log collection from service execution using an out-of-process daemon or sidecar with local disk spooling to avoid blocking production applications during network issues.

Hint 2

Full-text indexing every token in 5 GB/s of data causes massive write amplification; evaluate indexing only key metadata tags (service, level, host, trace ID) while keeping the raw message compressed.

Hint 3

Think about how partition keys in your streaming buffer (e.g., Kafka) should be organized—partitioning purely by service ID may cause hot partition issues if a single service is logging disproportionately.

Follow-up questions

  • How would you prevent a single misconfigured service generating 10x its usual log volume (a 'noisy neighbor') from degrading the entire cluster?
  • How would you design the system to comply with strict privacy policies such as PII/secrets redaction before logs hit permanent storage?
  • If an engineer wants to run a historical analytical query spanning 60 days of cold storage, how do you prevent that query from starving real-time troubleshooting searches?