Build a Durable Key-Value Cache
DatabricksSoftware Engineer · OnsiteLast updated
This question tests the ability to design a crash-recoverable key-value store by implementing a write-ahead log (WAL) to ensure data durability and system integrity after an unexpected failure.
Durable Key-Value Cache
You are asked to design a simple in-memory key-value store that survives system crashes. The store should support put(key, value) and get(key) operations.
Because an in-memory map is lost when a process restarts, you must implement a Write-Ahead Log (WAL) on disk. Every put operation must be persisted to the log before the in-memory map is updated. When the system restarts, it must be able to reconstruct the map by replaying the log.
Requirements:
put(key: str, value: str): Persist the operation to a file and update the memory state.get(key: str) -> str: Return the value for the key from memory.recover(): On startup, scan the log file and rebuild the memory map.
Constraints:
- Assume
keyandvalueare strings. - Use a file-based log (e.g.,
store.log). - Implement a mechanism to detect and handle partial writes (e.g., a crash occurring in the middle of writing a log entry).
Example:
# Execution 1
cache = DurableCache("store.log")
cache.put("user_1", "alice")
cache.put("user_2", "bob")
# Process crashes
# Execution 2
cache = DurableCache("store.log")
print(cache.get("user_1")) # Output: "alice"
Considerations:
- How will you define the format of your log entries to ensure you can identify the start and end of a record?
- How do you verify the integrity of an entry (e.g., if the process crashed while writing)?
- What is the performance impact of writing to disk on every
put?
Hints
Hint 1
Use a structured log format where each entry includes the key, the value, and a checksum to verify integrity.
Hint 2
During recovery, ignore any log entries that are incomplete or fail the checksum validation.
Hint 3
Consider adding a header to each log entry that specifies the lengths of the key and value strings to make reading easier.
Follow-up questions
- How would you implement log compaction to prevent the log file from growing indefinitely?
- If the cache becomes very large, how could you optimize the recovery time so you don't have to replay the entire log on every restart?
- How would you modify your design to handle concurrent access from multiple threads?
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.