Katkat
Back

Design a Security Monitoring Framework

GoogleSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSystem Design

Last updated

Design a scalable, low-overhead security monitoring framework capable of capturing OS-level events via eBPF across a massive cloud fleet, streaming telemetry, and detecting malicious activity in near-real-time.

Scenario

Large-scale cloud platforms run millions of workloads across heterogeneous fleets of virtual machines and bare-metal nodes. Security engineers need an automated framework to detect host compromise, unauthorized privilege escalations, lateral movement, and data exfiltration in real time without degrading host performance or leaking sensitive workload data.

Design an end-to-end security monitoring framework for cloud infrastructure, covering host-level observability, ingestion, stream processing, and alert detection.

Requirements

Functional Requirements

  • OS-Level Telemetry: Collect process executions (execve), network socket activity (connect, accept), file access, and privilege escalation attempts.
  • Local Processing: Filter known benign events, enrich events with container/cgroup and workload metadata, and securely batch events for transmission.
  • Detection Engine: Support both deterministic rule-based matching (e.g., unexpected shell spawned by a web server) and stateful stream-based sequence detection (e.g., port scan followed by credential file access).
  • Alerting & Audit Trail: Emit prioritized alerts for security operations teams and maintain a tamper-evident audit log for post-incident forensics.

Non-Functional Requirements

  • Host Performance: Host monitoring must consume strictly under 1.5% CPU and 150 MB RAM per host.
  • Scalability: Fleet of 500,000 nodes, generating an aggregate ingestion volume of up to 25 million events per second after host-level filtering.
  • Low Latency: High-severity detections must trigger alerts within 5 seconds of the event occurring on the host.
  • Tamper Resistance & Privacy: Ensure logs are resilient against host-level root compromises where possible, and sanitize credentials/PII before egress.

What Your Solution Should Cover

  1. Host-Level Architecture: Mechanism for capturing kernel events (e.g., eBPF vs. auditd, perf buffers vs. BPF ring buffers), user-space agent design, and local drop/backpressure strategies.
  2. Data Pipeline & Ingestion: Ingestion tier, transport protocols, buffering, and horizontal scaling strategy.
  3. Detection & Processing Engine: Architecture for processing high-throughput streaming events, tracking state across distributed nodes, and evaluating detection rules.
  4. Storage & Data Lifecycle: Tiered storage architecture for real-time querying (hot) vs. compliance and forensic analysis (cold).
  5. Security, Privacy, & Trade-offs: Masking sensitive command-line parameters/environment variables and defending the telemetry agent against evasion or termination.

Hints

Hint 1

Consider using eBPF with BPF ring buffers instead of older mechanisms like auditd to avoid excessive context-switching overhead and kernel lock contention.

Hint 2

Perform deduplication, benign-event filtering, and metadata enrichment (container ID, pod name) in the host user-space daemon before emitting telemetry over the network.

Hint 3

Design a tiered ring-buffer and backpressure strategy on the host to ensure system stability when network ingestion endpoints are degraded.

Follow-up questions

  • If an attacker gains root privileges on a monitored host, how can you prevent or detect the attacker unloading the eBPF probes or killing the user-space daemon?
  • How would you implement PII/secret scrubbing on command-line arguments and environment variables at wire speed without introducing major latency?
  • How does your stream processing layer handle out-of-order events when correlating multi-step attack chains across distinct hosts?