Katkat
Back

Design an encrypted log collection system

CloudflareSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

This question tests the ability to design a distributed, high-throughput ingestion pipeline that ensures log confidentiality and integrity using encryption at rest and in transit.

System Design: Encrypted Log Collection

Cloudflare operates a global network with hundreds of points of presence (PoPs). Design a distributed system that collects security and application logs from these PoPs, encrypts them, and aggregates them into a centralized cold storage system for long-term auditability.

Requirements

  • Functional: Collect JSON-formatted logs from local agents in each PoP, aggregate them, and store them in a durable, queryable data lake.
  • Security: Logs must be encrypted at the source (in the PoP) before hitting the network. Unauthorized access to the central aggregation servers should not expose plaintext logs. Implement a mechanism for key rotation.
  • Scale: The system must handle 500,000 log events per second across the global infrastructure.
  • Reliability: Ensure minimal data loss during transient network partitions.

Scope

Your design should address:

  1. Key Management: How are encryption keys generated, distributed, and rotated without decrypting logs at the aggregator level?
  2. Architecture: Describe the flow from log agent to cold storage, including any buffering or queueing layers.
  3. Data Integrity: How do you ensure logs haven't been tampered with or dropped during transit?
  4. Performance: How to minimize the latency impact on the edge servers generating the logs.

Constraints

  • Use standard protocols where possible.
  • Assume you have access to a distributed key-value store (e.g., etcd or Consul) for coordination.

Hints

Hint 1

Consider using an envelope encryption strategy where each batch of logs is encrypted with a unique data encryption key (DEK), which is then wrapped by a master key.

Hint 2

Introduce an asynchronous buffering layer (like Kafka or a distributed queue) to decouple the high-throughput producers at the edge from the slower processing/storage services.

Hint 3

For integrity, consider adding a cryptographic hash or HMAC to each log batch to detect modification before the data is committed to storage.

Follow-up questions

  • How would you implement a 'Search' functionality on these encrypted logs without decrypting them globally?
  • If a single PoP loses connectivity to the central aggregation server, how would you handle local backpressure to prevent the edge services from crashing?
  • What observability metrics would you track to detect a failure in the key rotation pipeline?