Design an Audit Logs Service

Scenario

Design a centralized, multi-tenant audit logging service for a cloud data platform. Every action performed across the platform—such as user authentications, privilege changes, query executions, and data export events—must be recorded immutably. Platform tenants must be able to search and export their own audit trails for compliance (e.g., SOC 2, HIPAA, GDPR) and security investigations.

Requirements

Functional Requirements

• Event Ingestion: Accept structured audit events asynchronously with low caller overhead and near-real-time visibility (within 30 seconds).
• Tenant-Scoped Search: Allow authenticated tenant administrators to query audit logs filtered by time range, actor ID, action type, and resource identifier.
• Immutability & Tamper-Evidence: Ensure that once written, logs cannot be modified or prematurely deleted by tenants or standard internal platform operations. Provide a mechanism to verify log integrity.
• Export / Streaming: Enable tenants to export filtered logs in batches (e.g., S3/GCS) or stream events to external SIEM systems.
• Retention Management: Enforce configurable retention windows per tenant (default 90 days online, up to 7 years in cold archive).

Non-Functional Requirements

• Durability: Zero data loss for acknowledged events (99.999999999% durability target).
• Isolation: Strict tenant boundary enforcement; cross-tenant data leaks must be impossible.
• Low Read Latency: P95 query response time under 2 seconds for searches spanning up to 30 days.
• High Availability: 99.99% availability for ingestion.

Scale Assumptions

• Tenants: 50,000 active organizations.
• Write Volume: 1 billion events/day (~12,000 writes/sec average, 50,000 peak writes/sec).
• Event Size: Average 1 KB per JSON record (~1 TB/day raw uncompressed data).
• Read Volume: Relatively low read-to-write ratio (~100 QPS across all tenants), but queries can scan large time ranges.

What to Cover

• API Design: Ingestion payload structure, search query API, and integrity verification endpoint.
• Architecture & Ingestion Pipeline: Decoupling ingestion from processing using durable buffers (e.g., message queues), stream processors, and storage sinks.
• Data Storage & Indexing: Schema design and storage tiering (hot search indexes vs. cold object storage) optimized for tenant-isolated time-series queries.
• Tamper-Evident Mechanisms: Cryptographic chaining (e.g., hash chains or Merkle trees) and cloud WORM (Write Once, Read Many) policies.
• Trade-offs: Latency vs. batching efficiency, search index cost vs. full-table scan trade-offs, and noisy-neighbor mitigation.