Implement a Recoverable In-Memory Key-Value Store
OpenAISoftware Engineer · Technical ScreenLast updated
This problem tests your understanding of durability, logging, and data persistence in an in-memory system.
Problem Statement
Design a thread-safe, in-memory key-value store that supports basic get(key) and put(key, value) operations. To ensure durability, the system must append every put operation to a local file (an Write-Ahead Log or WAL) before modifying the in-memory state. Upon process restart, the system must reconstruct the latest state by replaying the log.
Requirements
- Thread Safety: Multiple threads may call
getandputsimultaneously.getoperations should return the latest committed value. - Persistence: Ensure that a
putis safely flushed to disk. You must handle the scenario where the system crashes mid-write (torn writes). - Recovery: Implement a
load()function that reads the WAL and populates the in-memory store. - Data Format: Keys and values are arbitrary byte arrays.
Interface
class PersistentKV:
def __init__(self, log_path: str):
"""Initialize the store and recover from log_path if it exists."""
def put(self, key: bytes, value: bytes) -> None:
"""Store key-value pair and ensure durability."""
def get(self, key: bytes) -> bytes:
"""Retrieve the value associated with key."""
Example
If the log contains:
[Length: 3][Key: 'foo'][Length: 3][Val: 'bar']
[Length: 3][Key: 'foo'][Length: 3][Val: 'baz']
After load(), get('foo') should return b'baz'.
Constraints
- The log file grows indefinitely; focus on the recovery logic first.
- Handle potential partial writes at the end of the file caused by a crash.
Hints
Hint 1
Consider adding a checksum (e.g., CRC32) to each log entry to detect and ignore torn or corrupted writes during recovery.
Hint 2
Use a file lock or a mutex to ensure that concurrent 'put' operations do not interleave their entries in the log file.
Hint 3
For recovery, read the file sequentially and maintain a hash map; subsequent writes to the same key should simply overwrite the previous entry in your map.
Follow-up questions
- The log file grows indefinitely. How would you implement a compaction process to shrink the log size?
- How does your implementation handle system performance if the log file becomes several gigabytes large?
- If you needed to support atomic multi-key updates (transactions), what changes would you make to your log format and recovery logic?
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.