Design a File Storage and Sync Service

This is the Dropbox or Google Drive question. You save a file on your laptop, and a few seconds later it is on your phone and on your friend's computer. It sounds like "upload a file, download a file". The real work is in what happens when the file is big, when two people edit it at once, and when 50 million users do this all day.

1. Problem Statement

Design a service where a user keeps a folder of files in the cloud, and every device they own shows the same folder, kept up to date automatically. Users can share folders with other people, see old versions of a file, and recover from mistakes.

A good picture is a shared notice board in an office. Anyone can pin a new sheet or replace one, and everyone who walks by sees the latest. The hard part is when two people replace the same sheet at the same moment, and when the sheet is a 20 GB video.

2. Questions to Ask First

3. Requirements

3.1 Functional Requirements

3.2 Non Functional Requirements

1. Never lose a file. This is the one promise the product makes. Data must survive disk, machine and even data centre failures.
2. Sync should feel quick. Under 5 seconds from save to appearing on another online device is a good target.
3. Bytes must not be wasted. Dropped connections, tiny edits and repeated uploads of the same file should cost little.
4. Strong consistency for the file tree. If a device says "your folder is at version 812", that must be true. Showing a half finished upload as a complete file is a bug.
5. Highly available. Reads should keep working during partial outages.
6. Scales by adding machines, not by buying a bigger one.

4. Back of the Envelope Numbers

4.1 Storage

Logical data = 50M users x 4 GB = 200M GB = 200 PB
Files        = 50M users x 1,000 = 50 billion files
Average file = 200 PB / 50B = 4 MB

Deduplication (storing identical pieces once, explained in section 5) saves some of that. Say 25 percent, which is a cautious number for a consumer service where lots of people hold the same PDFs, installers and songs.

Stored after dedup = 200 PB x 0.75 = 150 PB
Saved              = 50 PB

4.2 Chunks

Each file is cut into chunks of 4 MB (section 9.1 explains why 4 MB).

Chunks (before dedup)  = 200 PB / 4 MB = 200e15 / 4e6 = 50 billion
Unique chunks (75%)    ~ 37.5 billion
Chunk index row        ~ 100 bytes (hash, size, location)
Chunk index size       = 37.5B x 100 B ~ 3.75 TB

Small files are one chunk that is less than 4 MB, so the true count is a bit higher. The shape is right.

4.3 Metadata

Metadata means the information about files, not the bytes inside them: names, folders, versions, which chunks belong to which file.

files          = 50B rows x 300 B   = 15 TB
file versions  = 3 kept on average
               = 150B rows x 150 B  = 22.5 TB
version_chunks = 150B rows x 40 B   = 6 TB
Total metadata ~ 45 TB

A good database shard holds about 500 GB comfortably, so 45 TB / 500 GB is roughly 90. We plan for about 100 shards.

Notice the ratio: 45 TB of metadata against 150 PB of data. Metadata is about 3,000 times smaller than the bytes, but every single user action goes through it. It is small and extremely busy. That is why the two are stored in completely different systems.

4.4 Traffic

Changes per day   = 10M active users x 20 = 200M
Per second        = 200M / 86,400 ~ 2,300
Peak (3x)         ~ 7,000 file commits per second

Data uploaded per change ~ 2 MB (many small files, some single changed chunks)
Per day   = 200M x 2 MB = 400 TB
Per second = 400 TB / 86,400 ~ 4.6 GB/s ~ 37 Gbps
Peak (3x) ~ 110 Gbps

Each change must reach about 1.5 other devices, so downloads are 200M x 1.5 x 2 MB = 600 TB a day, about 7 GB/s. Uploads and downloads are about the same size. This is not a "read heavy" system like a video site. Files are written, read once or twice by the user's own devices, then mostly sit still.

4.5 Metadata Query Rate

Each commit does about 5 database operations (read the file, insert a version, insert chunk links, update the file, add a journal entry).

Peak is 7,000 commits/s x 5 = 35,000 operations a second. Across 100 shards that is 350 per shard, which is easy. We do not need exotic technology for the file tree, we need good sharding and good indexes.

4.6 Online Connections for Sync

Say half of the 20M devices are online at the same moment and each holds one open "tell me when something changes" request.

Open connections = 10M
Each lasts ~ 60 seconds, then reconnects = 10M / 60 ~ 167,000 new requests per second
One server holds ~ 100,000 idle connections, so 10M / 100K = 100 servers

Nothing is scary here, but this fleet exists, separate from the API servers, since holding connections open is a different job from answering requests.

4.7 Money

Rough numbers: $20 per TB per month for object storage, $0.05 per GB for outgoing data at list price.

Storage = 150 PB = 150,000 TB x $20 = $3M per month (dedup saves 50,000 TB x $20 = $1M)
Download = 600 TB per day = 600,000 GB x $0.05 = $30K per day ~ $0.9M per month

Compare with a video site, where bandwidth is the huge bill. Here storage is the biggest bill, so the money tricks are about storing less: dedup, only keeping changed chunks of old versions, and moving untouched files to a cheaper cold tier. Say this contrast out loud, because it shows you know the cost drivers differ.

5. High Level Design

Three ideas drive the whole thing. Learn these and the rest follows.

5.1 Split Files Into Chunks

Never treat a file as one blob. Cut it into fixed pieces of 4 MB called chunks. A 1 GB file is 256 chunks. A 10 KB note is one small chunk.

Why bother? Think about editing one line in a 1 GB video project file. Without chunks, you upload 1 GB again. With chunks, one chunk changed, so you upload 4 MB. That is 256 times less. Chunks also give us resumable uploads, for free, as section 10 shows.

5.2 Name Each Chunk by Its Fingerprint

For every chunk, compute a hash. A hash (here SHA-256) is a fixed length fingerprint of the bytes: the same bytes always give the same fingerprint, and different bytes give a different one, for all practical purposes. Think of it as an ISBN number for a book. If two people have a book with the same ISBN, it is the same book, and you do not need two copies on the shelf.

So the chunk's name in storage is its hash. Before uploading anything, the client sends the list of hashes to the server and asks "which of these do you not have yet?". The server answers with the missing ones, and only those are uploaded. This one trick gives three wins:

1. Edit a big file: only the changed chunks are new.
2. Two files, or two users, with identical content: stored once.
3. Retry after a crash: the chunks already uploaded are simply "already have".

5.3 Keep Metadata and Bytes in Different Places

The bytes go to blob storage, an object store built for huge numbers of large, never edited files. The file tree goes to a database, built for small, constantly changing, carefully consistent facts. They have almost nothing in common, so mixing them hurts both.

A file in the system is nothing but a row in the database that says "this file, at this version, is made of these chunks, in this order". The chunks sit in the blob store. Reading a file means reading its row, then fetching its chunks.

5.4 The Diagram

5.5 Walking Through Each Box

Client. The app on each device is cleverer than it looks. It watches the folder for changes, cuts files into chunks, computes hashes, keeps a small local database of what it last synced, uploads, downloads, and puts files back together. Much of the logic lives here, so the servers can stay simple.

API servers. Answer metadata questions: which chunks are missing, commit this file, list changes, restore version. They never carry file bytes. A client uploads a chunk straight to blob storage with a short lived signed address. That keeps 37 Gbps of upload traffic off the API fleet.

Blob storage. An object store holding chunks. The key is the hash, for example chunks/ab/cd/abcd1234.... A chunk is never changed, only written once and later deleted, which makes the store simple and very safe: if the same chunk is uploaded twice, the second write puts the exact same bytes in the exact same place.

Chunk index. A fast key value lookup from hash to "yes we have it, here is its size and location". This answers "which of these 40 hashes are missing?" in a few milliseconds. About 3.75 TB, sharded by hash.

Metadata DB. The file tree: folders, files, versions, who has access. Sharded by namespace, which is the idea in section 7. It is the source of truth for what the folder looks like.

Message bus and notification servers. When a commit succeeds, a tiny event goes on a bus. Notification servers, which hold the open connections to devices, tell the right devices "something changed". They carry no file data, just a nudge.

Cleanup job. Deletes chunks that no version refers to any more. Done slowly and carefully, since deleting data is the one action you cannot undo.

6. API Design

6.1 Upload and Commit

Prepare:

{
  "namespaceId": "ns_9021",
  "chunks": [
    { "hash": "9f2c...e1", "size": 4194304 },
    { "hash": "77ab...03", "size": 4194304 },
    { "hash": "c410...5d", "size": 1048576 }
  ]
}

Response:

{
  "missing": [
    { "hash": "77ab...03", "uploadUrl": "https://blob.example/chunks/77/ab/...?sig=...", "expiresInSeconds": 900 }
  ]
}

Only one of three chunks needs uploading. The other two already exist, perhaps from this same user's previous version, or from someone else's identical file.

Commit:

{
  "namespaceId": "ns_9021",
  "parentId": "f_5510",
  "name": "report.docx",
  "baseVersion": "v_8812",
  "chunks": ["9f2c...e1", "77ab...03", "c410...5d"],
  "sizeBytes": 9437184,
  "modifiedAt": "2026-10-10T09:14:00Z",
  "deviceId": "dev_laptop_1"
}

baseVersion is the version the user started editing from. The server uses it to spot conflicts (section 9.3). For a brand new file it is empty.

Response 200 OK:

{
  "fileId": "f_7701",
  "versionId": "v_8813",
  "journalSeq": 20417,
  "conflict": false
}

To keep small files cheap, a commit request can carry up to 100 files. Otherwise, a folder of 5,000 tiny notes would be 5,000 round trips.

6.2 Sync

Long poll request:

{ "cursors": { "ns_9021": 20417, "ns_5532": 880 } }

Response when something changed:

{ "changed": ["ns_9021"] }

Or after 60 quiet seconds: { "changed": [] } and the client asks again. Then the client calls the changes endpoint with its cursor and gets back a page of journal entries (each with a seq, an operation such as UPSERT or DELETE, the file id and the version id), plus a nextCursor and a hasMore flag.

6.3 Versions, Sharing, Links

7. Data Model

7.1 The Key Idea: Namespaces

A namespace is one tree of files that has one list of members. Every user gets a personal namespace. A shared folder becomes its own namespace, and the people it is shared with mount it into their own tree.

Why this matters: everything that must be consistent together (the files, their versions, the change log, the permissions) lives under one namespace id. We shard by namespace_id, so a commit touches exactly one database shard and can be a normal transaction. If we sharded by file id or by user, committing to a shared folder would need a transaction across shards, and that is where bugs and slowness come from.

The cost: one very busy shared folder (a whole company folder with 5,000 people) lives on one shard. That is acceptable because a single folder still has a limited write rate, and we can move big namespaces to dedicated shards.

7.2 Tables

namespaces

memberships (kept in a small separate store, sharded by user)

Permission check on every request is one lookup by (user_id, namespace_id), cached for a minute, and removed from the cache immediately when a member is removed.

files

There is a unique rule on (namespace_id, parent_id, name) for items that are not deleted, so two files cannot share one name in one folder.

Why parent_id and not a full path like /Work/2026/report.docx? Because of renames. If you store full paths, renaming /Work means rewriting the path on every file under it, which could be a million rows. With a parent pointer, renaming or moving a folder is one row update, no matter what is inside. That is why moving a huge folder is instant in a good cloud drive. The cost is that building a full path means walking up the parents, which the client and a small cache handle.

file_versions

Versions are never changed after they are written. A new edit is a new row.

version_chunks

Why a separate table and not a list stored inside the version row? A 50 GB file has 12,800 chunks. At 32 bytes per hash that is 400 KB for one row, which is too much for a normal row. As a table, the list can be read in pages.

journal

The journal is the heart of sync. A device only needs to remember one number per namespace: "I have seen everything up to seq 20417". It asks for entries after that number and applies them in order.

If a device was off for more than 90 days, its number is too old and the journal has been trimmed. The server says so, and the device does a full comparison of its folder against the tree. Rare, slower, but correct.

chunks (the index, a key value store sharded by hash)

There is no reference count on chunks. A count would need to be updated, in a different shard from the commit, on every commit and delete. That is a cross shard write on the hottest path, and counts can drift after crashes. We find unused chunks with a periodic sweep instead (section 10).

share_links

8. Key Flows

8.1 Saving an Edit

The order is the safety net. Chunks go up first, the commit comes last, and the commit is a single database transaction. Until the commit lands, nobody sees the file change. So a crash halfway through an upload leaves some extra unused chunks behind and never a half broken file.

The transaction, in plain SQL:

BEGIN;
SELECT head_version_id FROM files
  WHERE namespace_id = ? AND file_id = ? FOR UPDATE;
-- if head_version_id is not the client's baseVersion, take the conflict path
INSERT INTO file_versions (...) VALUES (...);
INSERT INTO version_chunks (...) VALUES (...), (...), (...);
UPDATE files SET head_version_id = ?, modified_at = ? WHERE namespace_id = ? AND file_id = ?;
UPDATE namespaces SET last_seq = last_seq + 1 WHERE namespace_id = ? RETURNING last_seq;
INSERT INTO journal (namespace_id, seq, file_id, op, version_id) VALUES (...);
COMMIT;

The FOR UPDATE lock on the file row is what makes two simultaneous commits take turns, so the second one sees the first one's result.

8.2 Another Device Picks It Up

The phone already held two of the three chunks from the previous version, so it downloads 4 MB, not 9 MB. The saving is the same as on upload, in the other direction.

8.3 Two Devices Edit the Same File

9. Deep Dives

9.1 How to Cut the File: Fixed Size or Content Based

The problem. We want a small edit to cause a small upload. How we choose chunk boundaries decides whether that works.

Option A: fixed size chunks. Every 4 MB, cut. Chunk 1 is bytes 0 to 4 MB, chunk 2 is 4 to 8 MB, and so on. Very simple, very fast to compute, works for editing in place or appending. The weakness: if you insert one byte at the start of a file, every byte after it moves by one, so every chunk changes its content and the whole file is uploaded again.

Option B: content defined chunking. Slide a small window over the file and compute a rolling hash. Whenever the hash matches a special pattern, cut there. The boundaries depend on the content, not on the byte position, so an insertion near the start only changes the chunks around it, and the later boundaries land at the same content as before. Chunk sizes vary, say between 1 MB and 8 MB with an average of 4 MB.

The choice for this design: fixed 4 MB. My reasoning, with an honest caveat.

Most large files that people sync are photos, videos, archives and installers. They are either not edited at all, or rewritten whole, or appended to. Insertion at the front of a big file is rare. Fixed chunks cost almost no CPU, which matters on a phone with a small battery. And the boundaries are easy to explain and debug.

The caveat: for things like a large database file or a virtual machine disk, content defined chunking wins clearly, and some backup tools use it for exactly that reason. If this product had a lot of those users, I would add option B in the desktop client as a later upgrade. The server does not care, because it only sees hashes and lengths.

Why 4 MB and not 1 MB or 64 MB? Let us count.

Chunk size    Chunks for 200 PB     Index size at 100 B/row
1 MB          200 billion           20 TB
4 MB          50 billion            5 TB
64 MB         3.1 billion           0.3 TB

Small chunks give finer deduplication and smaller re-uploads, but a bigger index and more requests per file. Large chunks make the index small, but a one byte edit re-uploads 64 MB, and a dropped connection wastes more. 4 MB is the usual middle. It is big enough that a 1 GB file is only 256 requests, small enough that a retry costs seconds.

One security point worth mentioning. If the server says "I already have that chunk, skip the upload", a cheater who knows only the hash of someone's private file could claim to have it, skip the upload, and then download it. The hash is enough to "prove" ownership. Two fixes: only dedup within one account (loses the cross user saving), or make the client prove it really has the bytes, by asking for the hash of a few random byte ranges of the chunk mixed with a one time number. For dedup across users I would take the proof route, and always have the server check that uploaded bytes really hash to the claimed name.

9.2 Keeping Devices in Sync: Polling, Long Polling or Push

The problem. When one device commits a change, all the user's other online devices should know within seconds. A device cannot receive a call from the server unless it has a connection open for that.

Option A: poll every few seconds. The client asks "anything new?" every 30 seconds. Dead simple. But look at the cost:

10M devices online / 30 seconds = 333,000 requests per second
Nearly all of them answer "no, nothing new"
Average delay before a change shows up = 15 seconds

That is a lot of useless work and a slow feel.

Option B: long polling. The client asks "anything new since cursor 20417?" and the server does not answer until there is something, or 60 seconds pass. The request just waits. When the commit arrives, the server answers at once, and the client asks again.

Reconnects = 10M / 60 = 167,000 per second (half of A, and mostly useful)
Delay = the time it takes to forward a message, under a second

Option C: a WebSocket or similar always open connection. Similar memory cost per device, no reconnect cycle. Needs servers and every proxy and firewall in between to be happy with long lived upgrades. Some corporate networks are not.

The choice: B, long polling. It is ordinary HTTP that works through every proxy, it balances across servers like any other request, and the 60 second timeout doubles as a health check, since a dead connection is replaced within a minute. A WebSocket would be fine too, and I would not fight anyone who picked it. For phones in the background, skip the open connection entirely and use the phone's own push service (APNs or FCM) to wake the app.

Now, how does the right notification server learn about a commit? The simple way: every commit event goes to all 100 notification servers.

Peak commits = 7,000 per second
Each event ~ 50 bytes
Each notification server receives 7,000 x 50 B = 350 KB per second

That is tiny. Each server checks its own in memory map from namespace id to waiting requests, and wakes the ones that match. No complicated subscription routing. I would start here, and only add routing if the numbers grew by a factor of 100.

Two rules keep this reliable.

The notification carries no data. It means "go and look". If it is lost, nothing breaks, because the client re-polls within 60 seconds anyway, and it asks with its cursor, so it gets every missed change in order.

The cursor, not the notification, is the source of truth. The client always asks "everything after seq N". Whether it got zero, one or ten notifications in between does not matter. This one rule makes the whole system forgiving of lost, duplicated and late messages.

9.3 Conflicts: Two Devices, One File

The problem. The laptop and the phone both start from version 8813 of report.docx. The phone was in a tunnel. Each makes a change. Now the server has two different next versions, and both people worked hard on theirs.

Option A: last writer wins. Whichever commit arrives last replaces the other. Simple. Also quietly destroys someone's work, and the person never knows. For a service whose promise is "never lose a file", that is a broken promise.

Option B: keep both, as a conflicted copy. Use the baseVersion check: a commit is only accepted as the new head if its base version is the current head. If the server sees that the current head moved on, the later commit is saved as a separate file named report (conflicted copy from Phone, 2026-10-10).docx, in the same folder. Both versions exist, both sync to every device, and the human merges them by hand.

Option C: merge automatically. Works for text, and for documents inside an editor built for it (operational transforms or CRDTs, the machinery behind real time co-editing in Google Docs). The server would need to understand the file format, which is impossible for a general file store holding photos, zips and spreadsheets.

The choice: B. It never loses data, it needs no understanding of the file, and the rule is easy to explain: "if two devices edited the same file from the same starting point, you get two files". The people who dislike it are annoyed for ten seconds, which is far better than being angry about lost work for a week.

Details a good answer includes:

• The check happens inside the same transaction that takes the row lock, so two commits arriving at the same instant are put in order by the lock. The one that loses sees a changed head, and takes the conflict path.
• Retrying a commit must be safe. The client sends a unique request id, and if the same commit is sent twice (the response was lost), the server returns the first result and does not create two versions.
• Edit versus delete. One device deletes the file, another edits it from the old version. The edit wins: the file comes back with the edited content. Losing an edit is worse than an unwanted file.
• Two devices create the same new filename at once. Both have an empty baseVersion, and the unique name rule in the folder catches the second one, which becomes a conflicted copy.
• Rename versus edit is not a conflict at all, because moves change the files row and edits add a version. Both apply.

10. Failure Handling

The cleanup job deserves a closer look, because deleting data is where a storage service can do real harm. It runs weekly. It scans version_chunks to build the set of hashes still in use, and then deletes chunks that are not in that set and are older than 7 days. The 7 days protect chunks uploaded but not yet committed (a slow upload of a huge file might take hours or days to finish). It also runs a "dry run" mode first that only reports how many chunks it would delete, and an alert fires if that number is suspiciously large. A bug in this job is a data loss event, so it gets more care than almost anything else.

Retention works the same way. A job removes versions older than 30 days that are not the head, which removes their version_chunks rows, and the next sweep frees any chunks that nothing else uses.

11. Tradeoffs

12. Interview Talking Points

• Open with the questions: how many users, file sizes, how often files change, whether co-editing is needed. Then say you are building whole file sync, not live document editing.
• Say the three core ideas early and plainly: cut files into 4 MB chunks, name each chunk by its hash, and keep metadata in a database away from the bytes in blob storage. Everything else grows from these.
• Show the saving with a number: a 1 GB file with one edited chunk uploads 4 MB, not 1 GB. And a retry after a crash costs nothing, because the client just asks "which chunks do you still need?".
• Walk the commit order: chunks first, then one transaction that checks the base version, adds the version, moves the head pointer, bumps the sequence number and writes the journal. Say why sharding by namespace keeps that transaction on one shard.
• Explain sync with the cursor idea: each device remembers one number per namespace, long polling tells it when to ask, and lost notifications are harmless because the cursor is the truth.
• Handle conflicts with baseVersion. Name the three options (last writer wins, conflicted copy, merge) and say why you chose conflicted copies for a general file store.
• Cover sharing briefly: shared folders are their own namespaces with members and roles, links are random 128 bit tokens that can expire or be revoked.
• Close on cost and safety: storage is the biggest bill here (about $3M a month at 150 PB), dedup and keeping only changed chunks of old versions are the savings, and the cleanup job is carefully guarded with a grace period because deleting data is the one thing you cannot undo.