Katkat
Back

Design a KV store with QPS API

DatabricksSoftware Engineer · Technical Screen
HardSoftware EngineerTechnical ScreenSystem Design

Last updated

This question tests the ability to design a distributed key-value store architecture that incorporates observability, specifically tracking per-key or per-prefix Queries Per Second (QPS) metrics in real-time.

Problem Statement

Design a distributed Key-Value store that supports standard GET(key) and PUT(key, value) operations. The system must be highly available and partitionable (sharded). Additionally, the system must expose a GET_QPS(key_prefix) API that returns the number of requests per second for a given key prefix (e.g., "user:123:") over the last 60 seconds.

Functional Requirements

  • Support GET(key) and PUT(key, value).
  • Support GET_QPS(key_prefix) to retrieve real-time request volume metrics.
  • The system should be horizontally scalable.

Non-Functional Requirements

  • Low latency for KV operations.
  • High availability: The system should remain functional if a node fails.
  • Metrics accuracy: QPS tracking needs to be reasonably accurate (near-real-time is acceptable).

Design Constraints

  • Assume a scale of 100k requests per second across the cluster.
  • Assume keys are strings and values are binary blobs up to 1MB.
  • Explain how you handle data partitioning, replication, and the performance impact of monitoring metrics.

Hints

Hint 1

Consider using consistent hashing to shard keys across nodes for the KV store.

Hint 2

For QPS tracking, think about using a sliding window approach with a time-series data structure or an aggregate counter per shard.

Hint 3

To minimize impact on the critical path of GET/PUT operations, consider asynchronous metrics collection using a localized buffer or a sidecar process.

Follow-up questions

  • How would you handle a 'hot key' scenario where a single key receives 10,000 requests per second?
  • If the metrics requirement changes to 99th percentile latency tracking, how does your architecture evolve?
  • How would you ensure strong consistency for the KV operations if a node fails during a write?