Katkat
Back

Design a Weighted Log Distributor

Resolve.AiSoftware Engineer · Online Assessment
MediumSoftware EngineerOnline AssessmentSystem Design

Last updated

This question tests your ability to design a distributed system that routes incoming log streams to multiple storage clusters based on specified weight configurations while ensuring high availability and fault tolerance.

Problem Statement

Design a distributed log distribution service that ingests high-volume logs from various microservices and routes them to different storage backends (e.g., S3 buckets, Elasticsearch clusters, or local disks) based on dynamically configured weights. For example, you might need to route 70% of logs from Service A to Cluster X and 30% to Cluster Y.

Functional Requirements

  • Dynamic Weighting: The system must support updating distribution weights in real-time without service downtime.
  • Load Balancing: Distribute logs across clusters to prevent hotspots.
  • Fault Tolerance: If a destination cluster is unreachable, logs must be buffered or rerouted to a healthy node to prevent data loss.

Non-Functional Requirements

  • Throughput: The system should handle 100k events per second per node.
  • Latency: End-to-end latency for ingestion should be kept under 50ms.
  • Scalability: The architecture should scale horizontally as log volume increases.

Deliverables

  1. A high-level architecture diagram or description showing the ingestion, routing, and persistence flow.
  2. Choice of storage mechanism for the configuration weights.
  3. Strategy for handling destination backpressure or outages.
  4. API structure for managing routing rules.

Hints

Hint 1

Consider using a distributed message broker like Kafka as an intermediate buffer to decouple ingestion from processing.

Hint 2

Think about using a centralized configuration store like ZooKeeper or etcd to propagate weight changes to all routing instances efficiently.

Hint 3

Implement a circuit breaker pattern or a retry-with-exponential-backoff strategy to handle destination cluster health issues.

Follow-up questions

  • How would you ensure that log order is preserved if you need to route logs based on a hash of the 'user_id' rather than random weights?
  • If the log volume doubles unexpectedly, how would you implement auto-scaling for your ingestion tier?
  • How would you monitor the system to detect if a specific destination cluster is becoming a bottleneck?