Katkat
Back

Design a Distributed Logging System

UberSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

This problem evaluates the ability to design a high-throughput, fault-tolerant logging infrastructure capable of ingesting and storing massive volumes of telemetry data from diverse distributed services.

Distributed Logging System Design

Design a centralized logging system for a large-scale ride-sharing platform. The system must collect logs from thousands of microservices, process them in real-time, and store them for long-term analysis and debugging.

Requirements

Functional Requirements:

  • Ingestion: Services send logs via an agent or SDK.
  • Search/Query: Developers can search logs by service name, timestamp, and metadata (e.g., user_id).
  • Retention: Logs are kept in "hot" storage for 7 days and moved to "cold" storage for up to 1 year.

Non-Functional Requirements:

  • Scale: Handle 500,000 log events per second (avg) with spikes up to 2 million per second.
  • Latency: End-to-end latency from generation to searchability should be under 5 seconds.
  • Durability: Near-zero data loss is acceptable, though trade-offs are expected.

Expectations

Provide a high-level architecture diagram (described in text) covering:

  1. Data Pipeline: How logs move from the producer to storage.
  2. Storage Strategy: Choices for hot and cold storage tiers.
  3. Scalability & Reliability: How you handle traffic spikes and service failures.
  4. Data Model: How to index logs for efficient searching.

Hints

Hint 1

Consider using a distributed message queue like Kafka to act as a buffer between log producers and consumers to handle traffic spikes.

Hint 2

Think about the difference between unstructured logs (text blobs) and structured logs (JSON) and how this impacts indexing performance.

Hint 3

For long-term storage, explore partitioning data by time or service to optimize query performance in cold storage.

Follow-up questions

  • How would you handle a scenario where the logging system's storage backend becomes a bottleneck during a peak traffic event?
  • If logs contain PII (Personally Identifiable Information), what mechanisms would you implement to ensure security and compliance (e.g., GDPR) at the pipeline level?
  • How would you implement log sampling if the cost of ingesting 100% of logs exceeds the budget?