Diagnose I/O, Memory, and CPU Constraints

Scenario

You maintain a real-time market data feed handler and order execution engine written in C++ that processes millions of updates per second. During benchmark testing under simulated exchange peak loads, your service exhibits severe tail latency spikes across three distinct configurations.

For each of the three constraint categories below, provide a detailed diagnostic plan and resolution:

1. CPU Bottleneck: The service fails to keep up with incoming market data bursts, resulting in unbounded input queue growth despite low disk and network utilization.
2. Memory Bottleneck: The process suffers throughput collapse when maintaining a large stateful limit order book over long-running sessions, despite aggregate memory remaining below physical RAM capacity.
3. I/O Bottleneck: Order acknowledgment latencies degrade by an order of magnitude when writing audit log records during high-volatility market events.

Task

For each of the three scenarios, detail:

1. Diagnosis & Metrics: Which operating system tools (top, perf, iostat, vmstat, eBPF, etc.) and specific hardware/kernel counters (e.g., IPC, LLC cache misses, page faults, context switches, disk wait) would you inspect to confirm the root cause?
2. Root Cause Analysis: Propose a concrete architectural or code-level failure mode that matches the symptoms.
3. Optimization & Remediation: Describe the specific architectural, algorithmic, or systems-level change implemented to alleviate the bottleneck.
4. Verification & Correctness: How do you verify that the bottleneck is removed without violating functional correctness (e.g., data durability, thread safety, or order execution determinism)?