Designing PasteBin and Github Gists

The whiteboard for this problem is in excalidraw/7. Designing PasteBin and Github gists.excalidraw. This file is the full write up of that board.

1. Problem Statement

PasteBin is an online platform that lets you:

• Store any text file or code snippet, public to anyone.
• Share it with anyone, publicly or through a secret link.
• Set an expiration so it is deleted automatically.
• Get a unique URL for every paste.
• Store files that are long but not huge, up to 10 MB.
• Edit a paste if you created it.

2. Requirements

2.1 Functional Requirements

2.2 Non Functional Requirements

1. Read heavy. Pastes are read far more often than written, roughly 50 reads per write.
2. Durable. A paste must not be lost. Someone's shared snippet is the whole value of the product.
3. Large payloads. Content up to 10 MB means the content path and the metadata path have to be designed separately.
4. Reasonable read latency. Hundreds of milliseconds are acceptable. This is not a redirect, so we do not need single digit milliseconds.
5. Cheap. Bandwidth dominates the bill, so the design has to respect that.

2.3 Out of Scope

Comments, forking, revision diffs, private teams, code execution.

3. Where Should the File Live

Two options: a database, or a blob store such as S3. Crunch the numbers before choosing.

max size per file        = 10 MB
writes per month         = 10M

total storage per month  = 10M x 10 MB
                         = 10^7 x 10^7 B
                         = 10^14 B
                         = 100 TB per month

write to read ratio      = 1 to 50
read bandwidth per month = 100 TB x 50 = 5 PB per month

Two conclusions:

1. The data is huge and each file is large. There is no reason to put this in a database. Databases charge you in memory, backup time and replication for bytes they never query.
2. Bandwidth, at five petabytes a month, is the real cost driver. That points at object storage plus a CDN, both of which are built for exactly this.

Note that 100 TB per month is the worst case, since it assumes every paste is the full 10 MB. Real pastes average a few kilobytes, so the true figure is much smaller. The point of the calculation is that the ceiling rules out a database outright.

So: content goes to S3, metadata goes to a relational database.

Sizing the metadata:

uuid         36 B
name        120 B
created_at    4 B
visibility    4 B
owner_id      4 B
per row     168 B

10M x 168 B = 1,680 MB ~ 1.6 GB per month

One and a half gigabytes a month is easily handled by a single relational database, and it stays that way for years. A key value store such as DynamoDB works equally well, since every lookup is by paste id.

4. Architecture

4.1 API and Compute

For uploading and reading files we need API servers. One will not be enough, so we put many behind a load balancer.

4.2 The Bucket Layout

We configure the bucket so the path is fully determined by the ids we already have. When user 1729 uploads a paste with UUID 71293564:

s3://gist-paste/1729/71293564

The write path, step by step:

1. The user sends the file to an API server over HTTP POST.
2. The API server receives it.
3. The API server generates a random UUID.
4. The API server uploads the file to S3 at the derived path.
5. The API server writes a row in the meta database.
6. The API server returns the response.

The meta row for that example:

name is only what the UI displays as the file name.

We deliberately did not store the S3 path. It is s3://gist-paste/{owner_id}/{uuid}, which we can always build from the row we already have. Avoid storing anything that can be derived, because it means two places to keep in sync and one of them will eventually drift.

4.3 The Paste URL

The URL a reader visits is:

https://gist.github.com/71293564

The read flow is straightforward:

1. The request reaches an API server.
2. The server extracts the UUID from the path.
3. It reads the meta database row, which tells it the owner and the visibility.
4. It builds the S3 path from owner and uuid, and fetches the content.

5. Should We Cache

There will be some frequently accessed pastes, so it is tempting to cache them for lower perceived latency. We are for sure tempted. But is it worth it?

Start by defining the word. Frequently accessed, how much?

• A cache stores data in RAM, and 10 MB is enormous for a single cache entry.
• Cache memory is expensive per gigabyte compared to object storage.
• The read path already returns in a few hundred milliseconds, which is fine for a document view.

If the access frequency is not genuinely huge, caching does not make sense. As a one word answer: no.

I would rather serve ten requests from S3 than add an expensive cache and carry the cost of maintaining it. But stay open to it: if analytics show a small set of pastes taking an outsized share of traffic, revisit the decision with real numbers.

The middle ground, and what a real deployment would actually do first, is a CDN in front of S3. That gives edge caching for hot objects without any application level cache to invalidate, and it is billed as bandwidth rather than RAM. Since paste content rarely changes, CDN caching is nearly free correctness wise.

6. Expiration

The owner can set an expiration, after which the paste becomes inaccessible and is eventually deleted.

Mechanism:

1. Add an expires_at column.
2. When a paste is accessed, check the expiry first. Past it, return 404. Otherwise fetch and return the content.
3. Run a simple cleanup job that removes expired pastes from the meta database and from S3.
4. Run it once a day.

Two layers on purpose. The read time check gives correctness immediately, so nobody ever reads an expired paste even one minute before the job runs. The job reclaims space, which is not urgent.

S3 lifecycle rules can also expire objects automatically, but the meta row still needs deleting, so the job stays.

7. Fault Tolerance

• Take periodic backups of the meta database and store them on S3.
• This covers disaster recovery and accidental data loss.

The content itself is already durable, because S3 replicates objects internally. The metadata is the fragile part, since without it a paste is an unreachable object with no owner and no expiry. That is why the backup targets the meta database.

8. Analytics

Whenever a file is accessed, the API server captures the request metadata and stores it in Elasticsearch for end user analytics. Elasticsearch is good out of the box for recent aggregations and visualization, and Kibana renders the dashboards.

Analytics data is huge, and we rarely query beyond six months, so an archiver moves older indices to S3 to save space.

9. API Design

9.1 Paste APIs

Create:

{
  "name": "nginx.conf",
  "content": "server {\n  listen 80;\n}\n",
  "language": "nginx",
  "visibility": "SECRET",
  "expiresIn": "7d"
}

Response 201 Created:

{
  "uuid": "71293564",
  "url": "https://gist.example.com/71293564",
  "rawUrl": "https://gist.example.com/71293564/raw",
  "name": "nginx.conf",
  "language": "nginx",
  "visibility": "SECRET",
  "sizeBytes": 31,
  "createdAt": "2026-08-19T10:31:00Z",
  "expiresAt": "2026-08-26T10:31:00Z"
}

Note the response returns the URLs but the database does not store them. They are built from the uuid at response time.

Read an expired paste:

{ "error": "not_found", "message": "This paste has expired or does not exist." }

The same 404 for expired and non existent is deliberate. Telling a caller that a secret paste used to exist here leaks information.

9.2 Large Content and Presigned Uploads

For content near the 10 MB limit, routing bytes through the API server wastes capacity. A better path:

The client uploads straight to S3 and the API server never touches the bytes. The commit call is what makes the paste visible, so an abandoned upload leaves an orphan object that a weekly job cleans up, and never a half visible paste.

For small pastes, which is most of them, the direct POST is simpler and stays.

10. Data Model

10.1 Tables

pastes

There is no s3_path column, because the path is derived.

users

paste_versions

Editing writes a new object at s3://gist-paste/{owner_id}/{uuid}/v{version} and a new row here, rather than overwriting. Immutable objects mean a reader mid fetch never sees a half replaced file, and rollback is just pointing pastes.version back one step.

access_logs (Elasticsearch, not relational)

10.2 Relationships

In words:

• One user creates many pastes, one paste has at most one owner. Anonymous pastes have none, which is why owner_id is nullable and the S3 prefix falls back to a shared anonymous folder.
• One paste has many versions, one version belongs to exactly one paste. The current version is the one recorded on the paste row.
• One paste is accessed many times, one access record belongs to one paste.
• One user owns the analytics for all their pastes, which is why the log store is sharded by owner id.

access_logs has no enforced foreign key to pastes, because it lives in a different store. The relationship is logical and the join happens at query time.

11. Key Flows

11.1 Creating a Paste

S3 first, then the metadata. If the order were reversed, a crash in between would leave a readable metadata row pointing at an object that does not exist. This way a crash leaves an orphan object, which a cleanup job removes and no reader ever sees.

11.2 Daily Cleanup

S3 deletion happens before the metadata deletion. Deleting metadata first would lose the owner id, and then the S3 path can no longer be derived, which leaves an object nobody can ever find or bill correctly. This is the one real cost of deriving the path instead of storing it, and it is worth naming.

12. Deep Dives

12.1 Secret Pastes Are Unguessable, Not Private

Secret visibility means the paste is unlisted, not access controlled. Anyone with the UUID can read it. A version 4 UUID has 122 random bits, so guessing one is not realistic, but sharing the link is the same as making it public.

If real privacy is needed, that is a different feature: an ACL table plus an auth check on read, and presigned S3 urls with short expiry so the content url cannot be passed around either.

12.2 Editing Without Breaking Readers

Edits write a new version object rather than overwriting. The metadata row's version is the pointer to the current one. Flipping that pointer is a single row update, so a reader either sees version 3 or version 4 and never a partially written object.

content_hash lets us skip the whole write when someone saves without changing anything, which is a common UI accident.

12.3 Bandwidth Is the Bill

At five petabytes of reads a month, egress is far more expensive than storage. Levers, in order of impact:

Compression is unusually effective here because the payload is always text.

12.4 Why Relational for Metadata

Every lookup is by uuid, so a key value store would work. Relational wins on small details: the cleanup job wants WHERE expires_at < now() with an index, the profile page wants WHERE owner_id = ? ORDER BY created_at DESC, and both are secondary access patterns that a plain key value store makes awkward. The data is 1.6 GB a month, so nothing about relational hurts here.

If write volume grew a hundred times, shard on uuid and move the secondary queries to a separate index. Not now.

13. Failure Handling

14. Tradeoffs

15. Interview Talking Points

• Do the storage math first. Ten million writes times 10 MB is 100 TB a month, and at a 1 to 50 read ratio that is 5 PB of read bandwidth. Those two numbers decide S3 versus database before you discuss anything else.
• Then do the metadata math. 1.6 GB a month is the sentence that justifies one ordinary relational database.
• Say clearly that the S3 path is derived and not stored, and give the reason: never store what you can compute. Then name the cost, which is that the delete order now matters.
• The cache answer here is no, and being able to defend a no is more impressive than adding Redis by reflex. Follow it with the CDN, which gets the benefit without the invalidation problem.
• Expiry needs both a read time check and a background job, and each one exists for a different reason.
• Mention that secret means unguessable, not private. Interviewers probe this.