Katkat

Implement a Recoverable In-Memory Key-Value Store

OpenAISoftware Engineer · Technical Screen

Last updated

This problem tests your understanding of durability, logging, and data persistence in an in-memory system.

Problem Statement

Design a thread-safe, in-memory key-value store that supports basic get(key) and put(key, value) operations. To ensure durability, the system must append every put operation to a local file (an Write-Ahead Log or WAL) before modifying the in-memory state. Upon process restart, the system must reconstruct the latest state by replaying the log.

Requirements

  1. Thread Safety: Multiple threads may call get and put simultaneously. get operations should return the latest committed value.
  2. Persistence: Ensure that a put is safely flushed to disk. You must handle the scenario where the system crashes mid-write (torn writes).
  3. Recovery: Implement a load() function that reads the WAL and populates the in-memory store.
  4. Data Format: Keys and values are arbitrary byte arrays.

Interface

class PersistentKV:
    def __init__(self, log_path: str):
        """Initialize the store and recover from log_path if it exists."""

    def put(self, key: bytes, value: bytes) -> None:
        """Store key-value pair and ensure durability."""

    def get(self, key: bytes) -> bytes:
        """Retrieve the value associated with key."""

Example

If the log contains: [Length: 3][Key: 'foo'][Length: 3][Val: 'bar'] [Length: 3][Key: 'foo'][Length: 3][Val: 'baz']

After load(), get('foo') should return b'baz'.

Constraints

  • The log file grows indefinitely; focus on the recovery logic first.
  • Handle potential partial writes at the end of the file caused by a crash.

Hints

Hint 1

Consider adding a checksum (e.g., CRC32) to each log entry to detect and ignore torn or corrupted writes during recovery.

Hint 2

Use a file lock or a mutex to ensure that concurrent 'put' operations do not interleave their entries in the log file.

Hint 3

For recovery, read the file sequentially and maintain a hash map; subsequent writes to the same key should simply overwrite the previous entry in your map.

Follow-up questions

  • The log file grows indefinitely. How would you implement a compaction process to shrink the log size?
  • How does your implementation handle system performance if the log file becomes several gigabytes large?
  • If you needed to support atomic multi-key updates (transactions), what changes would you make to your log format and recovery logic?