Design centralized log ingestion and query system
RipplingSoftware Engineer · Technical Screen
Last updated
This problem tests your ability to design a distributed log processing pipeline capable of handling high-throughput ingestion and efficient time-range retrieval.
System Design: Centralized Log Ingestion and Query Service
Design a centralized log management system that allows developers to stream, store, and query logs across a fleet of microservices.
Requirements
Functional Requirements:
- Ingestion: Services push logs via a common API or agent.
- Querying: Users can search logs by service name, timestamp range, and keyword/regex patterns.
- Retention: Logs are kept for 30 days.
Non-Functional Requirements:
- High Throughput: The system must handle 100,000 log lines per second.
- Low Latency: Log search queries should return results within a few seconds.
- Durability: No logs should be lost once accepted by the ingestion endpoint.
Scope
Design the architecture including the data ingestion flow, storage schema, indexing strategy, and retrieval mechanism. Explain how you would handle spikes in traffic and how you would ensure query efficiency at scale.
Hints
Hint 1
Consider decoupling the ingestion layer from the storage layer using a distributed message queue to handle traffic bursts.
Hint 2
Think about the trade-off between full-text indexing (high storage cost, fast search) versus filtering by metadata tags.
Hint 3
For long-term storage, consider a tiered architecture that moves older data to cheaper, colder storage solutions.
Follow-up questions
- How would you handle a scenario where a specific microservice experiences a 'log storm' and generates 100x its normal traffic?
- If users need to perform aggregations (e.g., count errors per minute), how would you modify the storage or indexing strategy?
- How would you implement multi-tenancy and ensure that logs from one service cannot be queried by unauthorized teams?
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.