Design an Object Storage Service With Buckets, Keys and Durable Blob Storage
WaymoSoftware Engineer · Onsite
Last updated
Design an object storage service where clients store, read, list and delete blobs by key inside buckets. Covers the API, separating metadata from data nodes, replication versus erasure coding, visibility on write, prefix listing, repair and garbage collection.
Design an object storage service. Clients create named buckets and, inside a bucket, store, read, list and delete objects: opaque blobs of bytes addressed by a key, with a small amount of metadata such as size, content type and a checksum.
The round was given only as "design object storage", so establishing the scope and the numbers is part of the exercise.
Clarifying Questions
- What is the object size distribution: mostly small files, very large files, or both? Is there a maximum object size?
- What total capacity and what read and write request rates should the design target, and in how many regions?
- What durability and availability are expected?
- What consistency is required: must a read immediately after a successful write return the new object, and must a new object appear in listings immediately?
- Which features are in scope: overwrite, versioning, range reads, multipart upload of large objects, access control?
Hints
Hint 1
Two kinds of data: An object is a large blob plus a small record describing it. Ask whether both belong in the same storage system.
Hint 2
Durability has a price: Compare what full copies and erasure coding cost in disk space, in repair traffic and in read latency before you pick one.
Follow-up questions
- Most objects turn out to be a few kilobytes. What changes in the data layout?
- A disk holding part of a popular object fails during a spike in reads. Walk through what happens.
- How do you list a bucket with billions of keys by prefix, page by page?
- How would you add cross-region replication, and what consistency would readers in the second region see?
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.