Katkat
Back

Design a Compliance Audit Log Platform

LegoraSite Reliability Engineer · Onsite
HardSite Reliability EngineerOnsiteSystem Design

Last updated

Design an immutable, multi-tenant compliance audit logging platform that supports canonical ingestion, cryptographic tamper evidence, gap detection, and efficient legal investigation queries at scale.

Scenario

Legora provides legal-tech software to thousands of corporate and government tenants across dozens of independently deployed microservices (e.g., contract execution, discovery search, litigation holds, user permission changes). Regulators and enterprise compliance officers require an immutable, comprehensive record of all user, API, and system actions.

As a Site Reliability Engineer, your task is to design a centralized, tenant-isolated audit log platform capable of reliably ingesting, validating, persisting, and querying compliance events while providing verifiable tamper evidence and automated gap detection.

Requirements

Functional Requirements

  • Canonical Schema & Ingestion: Ingest structured audit events from disparate services with validation, tenant isolation, and client-side retry/idempotency guarantees.
  • Tamper Evidence & Immutability: Guarantee that logs cannot be altered or silently deleted by internal operators or compromised service credentials. Auditors must be able to cryptographically verify ledger integrity.
  • Gap Detection & Replay: Automatically detect missing log sequences or silent delivery failures from producer services and orchestrate replay without creating duplicate audit records.
  • Investigation Query Engine: Support fast, filtered searches (by tenant, actor, action, resource ID, and time window) for incident forensics and compliance audits.
  • Data Privacy & Redaction: Handle sensitive legal data and privacy compliance (e.g., GDPR/CCPA "right to be forgotten") within an append-only architecture.

Non-Functional Requirements & SRE Concerns

  • Durability & Availability: 99.999% durability target with zero data loss. Ingestion path must remain available (99.99%) even during storage compaction or index rebuilds.
  • Scale: 50,000 tenants; 1 billion events per day (average ~12,000 events/sec, peak ~40,000 events/sec). Log retention of 7 years.
  • Query Performance: Investigation queries across the last 30 days must return in under 3 seconds; cold queries across multi-year data must complete within 30 seconds.
  • Blast Radius Mitigation: A high-volume tenant or compromised producer service must not degrade ingestion or query performance for other tenants.

Deliverables

  1. End-to-End Architecture: Ingestion pipeline, message transport, buffering, validation layer, and storage tiering (hot vs. cold).
  2. Tamper-Evidence & Data Model: Detail the cryptographic verification mechanism (e.g., Merkle trees, hash chains, external witness anchoring) and event schema.
  3. Reliability & Operational Control: Define SLIs/SLOs, gap detection mechanisms, backpressure handling, deduplication strategy, and secret/key management for cryptographic shredding.

Hints

Hint 1

Decouple ingestion buffering from long-term immutable storage using a durable write-ahead message log with partitioned client-generated sequence numbers.

Hint 2

To achieve cryptographic tamper-evidence without sacrificing horizontal scale, consider batching events into per-tenant Merkle trees and anchoring periodic root hashes to an immutable witness or ledger.

Hint 3

Reconcile immutable append-only logs with legal privacy requirements (like GDPR erasure) by using cryptographic erasure, where sensitive fields are encrypted with individual keys destroyed upon deletion.

Follow-up questions

  • If an upstream producer suffers an extended network partition and flushes millions of delayed, out-of-order events, how does your system detect gaps versus delayed deliveries?
  • How would you implement an automated verification scanner that continuously validates hash integrity across petabytes of cold audit logs without incurring massive cloud egress and compute costs?
  • How does your architecture handle tenant noisy-neighbor issues during both sudden bursty ingestion and computationally expensive investigation queries?