Design a Durable Concurrent Event Writer

Scenario

You are building a high-performance, in-process Write-Ahead Log (WAL) writer for an embedded storage engine. Thousands of application worker threads concurrently produce opaque byte records that must be appended to a single logical file on local NVMe storage with strong durability, ordering, and consistency guarantees.

Requirements

Functional Requirements

• Client API: Expose an append(record: bytes) -> WriteReceipt interface (synchronous or asynchronous via futures/callbacks) returning the record's persistent offset or Log Sequence Number (LSN).
• Ordering & Atomicity: Preserved total order of acknowledged writes; records must never be interleaved or partially readable. An individual record write must be all-or-nothing.
• Crash Recovery: If the process terminates abruptly or power fails, a recovery routine must scan the file, detect the exact valid tail, discard any torn writes, and make the log ready for subsequent appends.
• Lifecycle & Teardown: Support graceful shutdown by draining queued records, performing an atomic disk sync, and closing descriptors cleanly.

Non-Functional Requirements & Scale

• Concurrency: Up to 5,000 concurrent producer threads invoking append.
• Throughput: Sustain at least 50,000 appends/sec and 200 MB/sec on modern local SSDs.
• Durability SLA: Support configurable durability, including a group-commit mode guaranteeing persistence within at most 5 ms or when a batch reaches 1 MB.
• Resource Boundaries: Strict memory limit for buffered in-flight writes (e.g., max 64 MB buffered); apply deterministic backpressure when disk write throughput drops.

What Your Design Should Cover

1. Concurrency & Thread Coordination: How producers hand off data without lock convoying (e.g., ring buffers, concurrent queues, multi-producer single-consumer patterns).
2. Record Framing & Disk Layout: The on-disk binary format (headers, magic bytes, length prefixes, CRC/checksums, padding/alignment).
3. I/O Pipeline & Batching: Group-commit mechanics, using vectorized I/O (writev), and handling fdatasync/fsync amortization.
4. Backpressure & Disk Failure: Handling buffer saturation, slow disks, ENOSPC (disk full), and hardware I/O errors.
5. Recovery Algorithm: Step-by-step logic to locate corruption or torn writes at log tail during restart.
6. Operational Metrics: Critical health signals to expose (e.g., queue lag, sync latency, batch fill ratio).