Design Dropbox
Problem
Design a file hosting and sync service like Dropbox or Google Drive. Users install a desktop client that keeps a folder in sync across all their devices, and files are also available through a web and mobile app. Changing part of a large file should upload only what changed. Users can share files and folders with other users, and earlier versions of a file can be restored for 30 days. The service has 500 million users, 100 million daily active users, and stores an average of 20 GB per user.
- What is the maximum file size? Is a 50 GB video file in scope?
- How should we handle two devices editing the same file while offline?
- Is deduplication allowed across users, or only within one account (it has privacy implications)?
- How quickly must a change on one device appear on another: seconds or minutes?
- Do shared folders need fine-grained permissions (view only, edit)?
- Do we need to support selective sync, where some folders stay online only?
Items in bold are existing requirements. The rest are suggestions. Confirm with the interviewer whether they are relevant first.
- Sync a folder across all of a user's devices automatically.
- Upload and download files through desktop, web, and mobile clients.
- Upload only the changed parts of a modified file.
- Share files and folders with other users.
- Keep previous versions for 30 days and allow restore.
- Detect conflicting edits and keep both copies rather than overwriting.
- Resume interrupted uploads and downloads.
Items in bold are existing requirements. The rest are suggestions. Confirm with the interviewer whether they are relevant first.
- Store 500 million users times 20 GB, about 10 PB of logical data, reduced by deduplication to perhaps 6 PB physical, then replicated or erasure coded.
- Changes propagate to other online devices within 10 seconds.
- Durability of 11 nines: once an upload is confirmed, it is never lost.
- Each daily user changes about 10 files per day, which is 1 billion file changes per day, or about 12,000 metadata commits per second on average and 40,000 at peak.
- Upload traffic: if each change averages 2 MB of new blocks, that is about 2 PB per day, or 200 Gbps average.
- Metadata must be strongly consistent per user namespace, so devices never disagree about which version is current.
Clients split files into content-defined chunks, hash each chunk, and upload only the chunks the server does not already have to a block store keyed by hash. A file version is just an ordered list of chunk hashes, committed to a strongly consistent metadata service that appends an entry to the namespace's change journal. Other devices hold a long-poll or WebSocket notification connection, and when told their namespace changed they fetch new journal entries and download any chunks they lack.
Loading diagram...
Upload a change. The desktop client's file watcher notices a modified file. It splits the file into chunks using a rolling hash (average 4 MB), computes a SHA-256 for each, and asks the sync API to commit the new file version as a list of hashes. The API replies with the hashes it does not have. The client uploads only those chunks to the block service, then retries the commit, which now succeeds.
Commit. The metadata service checks that the client's base version matches the current version of the file. If it does, it writes the new version and appends a journal entry with the next cursor for the namespace. If not, the client has a conflict and saves its copy as a separate "conflicted copy" file.
Sync to other devices. Each device keeps a notification connection open. When the journal advances, the notification service tells the device "namespace 123 changed". The device calls GET /changes?cursor=..., receives the new entries, downloads any chunks it does not already have locally, and assembles the file.
Share a folder. The owner shares a folder, which becomes its own namespace mounted into each member's tree. Members' devices then follow that namespace's journal too.
Watches the local folder, chunks and hashes files, and reconciles local state with the server.
- Native file system watchers (FSEvents, inotify, ReadDirectoryChangesW).
- A local SQLite database recording the last synced state of every file and the namespace cursor.
Diagram
Loading diagram...
Tradeoffs
Content-defined chunking (cutting where a rolling hash matches a pattern) means inserting a byte near the start of a file shifts only one chunk boundary, so the rest of the chunks keep their hashes. Fixed-size chunks would shift every boundary and reupload the whole file. Writing downloaded files to a temporary path and renaming them avoids applications reading half-written files.
Stores the file tree, versions, and per-namespace change journal, and enforces consistency.
- Sharded MySQL (or a distributed SQL database) with transactions, sharded by namespace ID.
API
POST /commitwith{ namespaceId, path, baseVersion, chunkHashes, size }returns{ newVersion },{ missing: [...] }, or409 Conflict.GET /changes?namespaceId=&cursor=returns journal entries after the cursor.GET /files/:id/versionsandPOST /files/:id/restore.
Data Model
files:namespace_id,file_id,path,current_version,is_deleted.file_versions:file_id,version,chunk_hashes,size,modified_by,created_at.journal:namespace_id,cursor(monotonic per namespace),file_id,version,op.
Scalability
All operations on one namespace go to one shard, so each commit is a local transaction. 40,000 commits per second at peak spread across hundreds of shards. Hot namespaces (huge shared team folders) get dedicated shards.
Tradeoffs
Metadata needs strong consistency and transactions (a commit must check the base version and append to the journal atomically), so a relational store fits. Blocks are immutable and content addressed, so they can live in a much cheaper, eventually consistent store. Splitting the two lets each scale on its own terms.
Stores immutable chunks keyed by hash, and deduplicates them.
- Object storage (S3, or a custom erasure-coded store at Dropbox scale).
- A block index in a key-value store:
hash -> location, refcount.
API
PUT /blocks/:hashuploads a chunk; the server verifies the hash before storing.GET /blocks/:hashdownloads a chunk, usually via a short-lived signed URL.
Scalability
Uploads and downloads go straight to the block tier with signed URLs, so the API servers never carry file bytes. Erasure coding (for example 10 data plus 4 parity fragments) gives high durability at about 1.4 times raw size instead of 3 times for triple replication.
Tradeoffs
Deduplicating across all users saves the most space but lets an attacker learn whether someone else has a given file by watching whether an upload is skipped. Restricting deduplication to within one account (or one team) avoids that leak at the cost of less savings. Chunks are garbage collected when their reference count reaches zero after the 30 day version window.
Tells online devices that a namespace they follow has new changes.
- Long-poll HTTP or WebSocket servers holding millions of idle connections, subscribed to journal events through a pub/sub layer.
Scalability
Notifications carry only "namespace changed", never data, so messages are tiny. Devices then pull changes through the sync API at their own pace, which naturally rate limits catch-up after long disconnections.
Tradeoffs
Polling every device every few seconds would mean tens of millions of useless requests per second. A push hint plus pull keeps the server's change journal the single source of truth, so a lost notification is harmless: the device catches up on the next one.
Manages shared folders and their membership.
Data Model
namespaces(namespace_id, owner_id, type)where type is personal or shared.mounts(user_id, namespace_id, mount_path, role).
Tradeoffs
Modeling a shared folder as its own namespace means a change is written once and every member follows the same journal, instead of copying the change into each member's tree.