Katkat
Back

Explain storing files to disk with concurrency

DatabricksSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSoftware Engineering Fundamentals

Last updated

This question tests your understanding of OS-level file I/O operations, the risks of concurrent writes to the same file descriptor, and how to implement robust thread-safe mechanisms to maintain file integrity.

The Problem

You are building a logging utility for a distributed system. Multiple worker threads need to write structured audit logs to a single shared file on disk. The system must ensure that log entries do not overlap, are written atomically, and are persisted to disk to prevent data loss in the event of a system crash.

Requirements

  1. Design a thread-safe FileLogger class that exposes a log(String message) method.
  2. Ensure that even under high concurrency, log entries remain distinct and are not interleaved.
  3. Explain how you would handle the trade-off between performance (throughput) and durability (guaranteeing the data reaches physical media).

Implementation Context

Assume the target environment is a POSIX-compliant filesystem. You are using Java, but you may discuss low-level OS behaviors.

Example

If two threads call log("A") and log("B") simultaneously, the file should eventually contain:

[Timestamp] A
[Timestamp] B

It must NOT contain:

[Timestamp] AB

Considerations

  • Does the OS guarantee atomicity for write() calls larger than the pipe buffer size?
  • How does FileChannel.force(boolean metaData) or fsync() affect performance?
  • What happens if the process crashes after a write() call but before the OS flushes the page cache?

Hints

Hint 1

Consider whether you need a mutex/lock at the application level or if you can rely on the O_APPEND flag at the OS level.

Hint 2

Think about the difference between writing to the application buffer, the OS page cache, and the actual physical disk platter/SSD flash cells.

Hint 3

Evaluate if a single-threaded background writer (producer-consumer pattern) might be more efficient than blocking all threads on a shared lock.

Follow-up questions

  • How would your approach change if you needed to support rotating logs (e.g., creating a new file every 100MB)?
  • What are the implications of using memory-mapped files (mmap) for this use case compared to standard write system calls?
  • How does the choice of filesystem (e.g., ext4 vs. ZFS) impact your durability guarantees?