Design a Shared, Crash-Safe Telemetry Buffer for Thousands of Worker Threads

You're building telemetry collection for a long-running batch-processing service. A single process spins up roughly 4,000 worker threads, each one periodically reporting small status records — think progress counters and timing samples — by calling into one shared component:

class TelemetrySink:
    def __init__(self, path: str): ...
    def record(self, payload: bytes) -> None: ...

Each payload is at most 4 KB and opaque: the sink never needs to parse or interpret it. All 4,000 threads share one TelemetrySink instance, and every call appends payload to one append-only file on local disk. The interviewer opens by asking you to describe how you'd build record, and to drive the conversation with your own questions before settling on a design.

The service currently targets roughly 50,000 record calls per second in aggregate across all threads, with a p99 latency budget per call of under 10 milliseconds, even though an fsync on the target disk can take several milliseconds on its own. A nightly reconciliation job reads the file after the process exits (or after a crash) and must be able to recover every complete record, cope with a possibly-corrupt final record, and tell records apart without any out-of-band index.

What you should cover

• A precise durability contract: exactly what must be true about a record once record returns, and what happens to records written just before a crash.
• A concurrency design that lets many threads' calls share a small number of expensive disk operations, instead of paying one fsync per call.
• An on-disk record format that survives a crash mid-write: the reconciliation job must be able to walk the file, recover every intact record, and detect (not necessarily recover) a partially written final record.
• How backpressure works when callers produce records faster than the disk can absorb them, and what a slow or failing disk does to record's callers.
• Rough numbers: how big a batch, how long callers might wait, and how that interacts with the latency budget above.

Clarifying questions worth raising

• Does "durable" here mean safe against an OS crash or power loss, or is handing the bytes to the kernel's page cache enough?
• Do records from different threads need any particular relative ordering in the file, or just internal consistency per thread?
• Can record ever block the caller, and for how long, under normal load versus a struggling disk?
• Is the file rotated or size-bounded, and does the sink need an explicit shutdown/flush call?