Design a Cloud File Storage Service
HarveySoftware Engineer · Technical Screen
Last updated
This question evaluates competency in designing large-scale distributed file storage and collaboration systems, covering storage and metadata modeling, API design, synchronization and consistency models, resumable large-file handling, versioning, access control, scalability, availability, durability, and security.
Design a cloud file storage and collaboration service similar to Google Drive or Dropbox.
At a high level, the service must let users store files in a hierarchical folder structure, share them with fine-grained permissions, and keep multiple clients (web, desktop, mobile) in sync. Your design should cover the following capabilities:
- Upload, download, delete, move, and rename files and folders.
- A hierarchical folder structure scoped per user or per organization.
- Sharing files and folders with users or groups using roles such as owner, editor, commenter, and viewer, with inheritance down a folder subtree.
- Large-file uploads that are chunked and resumable (survive a dropped connection).
- File version history and the ability to restore a prior version.
- Synchronization across web, desktop, and mobile clients, including offline edits that reconcile on reconnect.
- Search by file name and basic metadata (owner, type, modified time).
- High availability, durability, and security.
Walk through the end-to-end design: high-level architecture, the public API surface, the split between metadata and blob storage, the metadata/data model, the upload and download paths, the synchronization protocol, the consistency model, the scaling approach, failure handling, and the major trade-offs you would call out.
Constraints & Assumptions
- Users: tens of millions of registered users; assume a few million daily-active clients each maintaining a persistent sync connection.
- Data: petabytes of stored bytes; individual files from a few KB to several GB.
- Traffic: read-heavy on download/list/sync (assume reads dominate writes by roughly an order of magnitude), but sustained write traffic from uploads and edits.
- Durability target: very high for file content (object stores commonly advertise ~11 nines of annual durability via cross-zone replication); plan to match that.
- Availability: the metadata and download paths must stay available; brief degradation of search or thumbnails is acceptable.
- Security: TLS in transit, encryption at rest, short-lived signed access to blobs, and an audit trail for share/download events.
- Out of scope unless raised: real-time co-editing of a single document (operational transforms / CRDTs) — treat that as a separate collaboration system; you only need last-writer / conflict-copy semantics for whole-file sync.
Clarifying Questions to Ask
- Is the unit of isolation a single user's drive, or are there shared organization/team drives with their own permission roots? This changes the sharding key.
- What is the expected file-size distribution and the largest file we must support? It drives the chunk size and whether we need multipart at all.
- Do we need block-level delta sync (Dropbox-style: only re-upload changed chunks of an edited file), or is whole-file re-upload acceptable for v1?
- How strong are the sharing/permission requirements — simple ACLs, or nested groups, public links, expiring links, and link passwords?
- Is server-side full-text search (indexing file contents, not just names) in scope, or only metadata search?
- What are the retention rules for deleted files and old versions (trash window, version count cap, legal hold)?
- Do we need multi-region active-active writes (global low-latency, e.g. for an enterprise customer like a law firm operating across offices), or is single-region-write with regional read replicas sufficient?
Hints
Hint 1
Where to start: The single most important decision is to **separate metadata from blob content**. Metadata (the namespace tree, permissions, versions) lives in a transactional store and is small; file bytes live in object storage and are large and immutable. Almost every other decision follows from this split.
Hint 2
Large files: For resumable uploads, think about splitting a file into fixed-size **chunks**, uploading each chunk independently (ideally direct-to-object-storage via pre-signed URLs), checksumming per chunk, and committing a **manifest** of chunk keys as an atomic "complete" step. Content-addressing chunks by hash gives you dedup and idempotent retries for free.
Hint 3
Sync: Don't have clients diff their whole tree on every poll. Maintain a **monotonic per-drive change log** (sequence numbers / cursor); a client sends its last cursor and receives an ordered delta. This is the backbone of both incremental sync and conflict detection.
Hint 4
Scale & consistency: Reason separately about the two stores. Metadata wants **strong consistency** within a drive (rename/move/permission must be transactional) → partition by drive/workspace so each shard is a single-writer transactional unit. Blobs want **durability + cheap reads** → replicated object storage + CDN for immutable versioned blobs. Push everything non-critical (search indexing, thumbnails, virus scan, notifications) onto an async event bus.
Follow-up questions
- A user **moves a 50,000-file folder** into a new parent that has different sharing. How do you make the move appear atomic, and how do you propagate the permission change without rewriting 50,000 rows synchronously?
- Two desktop clients edit the **same file while both are offline**, then both reconnect. Walk through exactly what your sync protocol does — how the conflict is detected and what the user sees.
- How would you add **block-level delta sync** so that editing one page of a 2 GB file re-uploads only a few chunks? What changes in the data model and the upload path?
- Storage cost is dominated by **duplicate content** across users (e.g. the same PDF shared widely). How do you deduplicate safely, and what are the privacy and security pitfalls of cross-user dedup?
- A whole region hosting the **metadata primary** goes down. What is your failover story, what is the RPO/RTO, and what could a client observe as lost or stale during the cutover?
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.