Design Pastebin

Design Pastebin. Start with one server and one table, let the numbers move the text to object storage and views to a CDN, then stress it with expiry, caching rules, a viral paste and abuse, with interview prompts and code.

Foundation29 min core · 46 min full depth

Read first: Estimation Caching URL shortener
Scaling patterns: Multi-region and active-active CQRS and materialized views
Building blocks: Blob storage CDNs Caching Partitioning Queues and streams Rate limiting Unique IDs
Related: URL shortener Dropbox / Google Drive

“Design Pastebin. Someone pastes a block of text, gets a link, and anyone with the link can read it.”

This sounds like a URL shortener with longer values. The longer values are the problem. A short link points to 100 bytes. A paste can be anything from one line to 10 MB. And most pastes delete themselves after a set time.

That splits the system in two. There is a small, busy store of metadata: does this paste exist, who may read it, and when does it expire? And there is a large, quiet store of bodies: the text itself. We start with one table that holds both, and let the numbers pull them apart. Then most of the page is about keeping the two in step: when a paste is created, when it expires, when it is deleted, and in every cache between them.

The answer in one minute

If you only have a minute, this is the design. The rest of the page explains each part.

Text goes in an object store, details go in a database. Bodies add up to 24 TB. Metadata is only 1.5 TB. Keeping them apart keeps the database small and fast.
On create, we save the body first and the database row second. If we crash between the two, we leave behind a body nobody can find, which a cleanup job removes. We never leave a row that points at nothing.
A CDN serves most views. A paste never changes, so a cached copy can’t be wrong. The CDN may keep it only until the paste expires. Private pastes never go on the CDN.
Every way a paste can die sets one field, expires_at. Deletes, takedowns and “burn after reading” all mean “expires now”. Every read checks that field. A background sweeper frees the bytes later.
Abuse checks happen in two places. Cheap checks run during the create. Deep scans run just after it, and risky pastes stay hidden until they pass.

Scope it

A few questions change this design a lot. The answers are the assumptions the rest of the page builds on.

Interviewerminute 0

Design Pastebin.

What they are testing

Whether you ask about size, expiry and who can read a paste before drawing anything.

These four questions matter most:

How big can a paste be? Up to 10 MB for accounts, 512 KB without one. The average is 20 KB, and half are under 2 KB. This answer decides where the text lives.
Do pastes expire? The owner picks from 10 minutes to a year, or never. 60% pick an expiry. This decides how bytes get deleted.
Who can read a paste? Three choices. Public pastes can be listed. Unlisted pastes can be read by anyone with the link. Private pastes can only be read by their owner. This decides what a CDN may cache.
Can a paste be edited? No. “Edit” saves a new paste. So a paste never changes, which makes caching safe.

There is one more option worth knowing about: burn after reading. The paste can be read exactly once, and then it’s gone.

From these answers we make four promises:

Views work 99.95% of the time, and take under 50 ms from a nearby CDN for 99% of views.
A saved paste is never lost before it expires.
An expired paste can’t be read anywhere, from the second it expires. A deleted paste is gone from our servers at once, from the CDN within a minute, and from browsers within 5 minutes.
Unlisted and private links can’t be guessed.

We leave out search, a public feed of new pastes, editing with history, and billing.

Go deeperMore questions, and what each changes1 min

Question	We assume	If the answer were different
How many pastes a day?	5 million	10 times more means splitting the database from day one
Accounts?	Optional. Anonymous pastes are allowed, with limits	Accounts only would make abuse much easier to control
A 64 KB size cap?	No, 10 MB	With a 64 KB cap, text could live in database rows, with no object store at all
Mostly private?	No, most pastes are public or unlisted	If most were private, the CDN would help very little

Users can also delete a paste, and anyone can report one. Moderators can take a paste down. The creator can read their paste the moment it is created.

The numbers that decide it

Now we work out how big this is. The numbers will tell us to keep text and details apart.

Assumptions

New pastes a day: 5 million
Views per paste: 20 to 1
Peak over average: 10×
Average paste size, raw: 20 KB
Compression (gzip): 3×
Pastes that never expire: 40%
Years to plan for: 5 years
CDN hit ratio: 85%

Writes58/s580/s at peak

5 million pastes a day ÷ 86,400 seconds ≈ 58 creates a second, and about 580 at a 10 times peak. Each paste is written once and deleted once.

So we one ordinary database is enough for metadata.

Reads1,200/s12,000/s at peak

About 20 views per paste gives 1,200 views a second. The peak comes from one paste linked on a big forum. Plan for 12,000 a second, much of it on a single paste.

So we we need a CDN, and a plan for one very popular paste.

Storage24 TBbodies; 1.5 TB metadata

Over 5 years, about 3.6 billion pastes are still alive. Each is 20 KB of text on average, and compression makes that about a third. Bodies: 3.6 billion × 20 KB ÷ 3 ≈ 24 TB. Metadata: 3.6 billion × 400 bytes ≈ 1.5 TB.

So we keep them apart. 1.5 TB of metadata fits one database. Adding 24 TB of text to it would make every backup and recovery about 17 times slower.

Cost$2,300a month

Storing 24 TB in S3 costs about $560 a month. But 5 million uploads a day cost $750 a month, more than the storage. And 20 TB a month of CDN traffic costs about $1,000.

So we count requests and traffic, not just bytes.

You can change the numbers in the calculator. At 10 times the pastes, metadata reaches 15 TB and we must split the database from day one. At 300 views per paste, peak traffic passes 1 GB a second, and only a CDN should serve it.

Change an assumption

New pastes a day5 millionViews per paste20 to 1Peak over average10×Average paste size, raw20 KBCompression (gzip)3×Pastes that never expire40%Years to plan for5 yearsCDN hit ratio85%

Creates, average: 58/s
Creates, peak: 580/s; One relational primary takes this, even counting the later delete of each paste.
Views, average: 1,200/s
Views, peak: 12,000/s; Past the 10,000 reads/s one database serves: a CDN and a metadata cache are required.
Views reaching the origin, peak: 1,700/s; What the app servers, the cache and the object store actually see.
Pastes created over the plan: 9.1 billion
Pastes still live at the end: 3.6 billion
Blob storage, compressed: 24 TB; Too big for a database row store, small for an object store.
Metadata, 400 bytes a paste: 1.5 TB; Fits one database node: partition later, not on day one.
Hot metadata to cache: 8 GB; Fits one cache node; replicate it for availability.
Egress at peak: 77 MB/s; Small for a CDN. Serve gzip as stored so nobody recompresses.
Egress a month: 20 TB; At a few cents a GB, egress costs about as much as storing every paste.
Storage bill at the end, $ a month: 560; Object storage list price, about $0.023 per GB-month.
PUT requests, $ a month: 750; About $0.005 per 1,000 PUTs. For small pastes, requests rival storage.

Build the design

We start with the simplest design that works. Then we let the numbers break it, one at a time. Each step fixes only what broke, adds only the boxes that fix needs, and says what the fix costs.

Step 1. One server, one table

The smallest design is one app server and one database table, with the text stored in the row. Creating a paste is one insert. Reading one is one select. It works for a small site.

Pastebin · architecture

Stage 1: One server, one table

Color keyServiceDatabaseCacheQueue or streamBackground workerNew in this stageAsync or occasional

Paste views per second: 12,000

Stage 1 of 6

One server, one table

Why this stage

Nothing yet.

This is the smallest design that meets every functional requirement.

What we add

An app server that stores each paste, text included, as one Postgres row, and reads it back by ID.

What it costs

Megabytes of text in database rows, a database read on every view, and no redundancy.

At 12k paste views per second, App servers is 400% busy. Stage 2 is the fix.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What one machine can do

One app server serves about 3,000 views a second of 20 KB pastes.
One Postgres node does about 10,000 point reads a second and 5,000 writes. While it also stores the paste bodies, it manages about half that.
One Redis node answers about 100,000 reads a second.
One scanner checks about 300 average pastes a second.

The object store

The object store spreads keys across partitions, each good for about 5,500 reads a second. Across our keys, that is about 200,000 a second.

What the caches save

The CDN answers about 85% of views, or 90% with an origin shield.
Redis answers about 95% of what is left.

There are two main calls. One creates a paste. The other reads its raw text:

POST /api/v1/pastes?expires=1d&visibility=unlisted   Idempotency-Key: 3b9f...   <the text>  → 201 { "id", "url", "rawUrl" }
GET  https://pbcontent.example/raw/aZ3kP9qXr2LmT8vB                                          → 200, the text

The raw text is served from a separate domain. A paste is text that anyone can write, and it may contain HTML or script. On its own domain, with no cookies, that code can never act as the logged-in user on the main site. GitHub does the same with raw.githubusercontent.com.

Here is the table we end with. There are two important choices in it. There is no column for the text, and step 2 explains why. And every way a paste can die sets expires_at:

CREATE TABLE pastes (
  id                VARCHAR(16) PRIMARY KEY,
  owner_id          BIGINT,                    -- empty: anonymous
  visibility        SMALLINT    NOT NULL,      -- 0 public, 1 unlisted, 2 private
  blob_key          TEXT        NOT NULL,      -- where the text is in the object store
  burn              BOOLEAN     NOT NULL DEFAULT false,
  burned_at         TIMESTAMPTZ,               -- set once, by the one reader who wins
  scan_status       SMALLINT    NOT NULL DEFAULT 0,  -- 0 pending, 1 clean, 2 blocked
  created_at        TIMESTAMPTZ NOT NULL DEFAULT now(),
  expires_at        TIMESTAMPTZ                -- empty: never. Delete, burn and takedown set it to now()
);
CREATE INDEX pastes_expiry ON pastes (expires_at) WHERE expires_at IS NOT NULL;

The numbers break this first design in two places: 24 TB of text in a database, and 12,000 views a second. The next steps fix them in that order.

Go deeperTwo kinds of ID1 min

A public paste’s ID only needs to be unique and short, so it uses the URL shortener’s shuffled counter, 8 characters. An unlisted or private paste’s ID is the only thing between the paste and a stranger. So it is a random secret: 16 characters, about 95 bits.

Go deeperThe full API1 min

POST /api/v1/pastes?expires=1d&visibility=unlisted&syntax=python
Authorization: Bearer <token>            (optional: anonymous creates get lower limits)
Idempotency-Key: 3b9f0c2e-...
Content-Type: text/plain; charset=utf-8

201 Created
{ "id": "aZ3kP9qXr2LmT8vB", "url": "https://pb.example/aZ3kP9qXr2LmT8vB",
  "rawUrl": "https://pbcontent.example/raw/aZ3kP9qXr2LmT8vB",
  "expiresAt": "2026-10-29T14:03:00Z", "deleteToken": "dt_..." }

DELETE /api/v1/pastes/{id}                 owner, or the delete token; 204
POST   /api/v1/pastes/{id}/raw-link        private: a signed raw URL that works for 60 s
POST   /api/v1/pastes/{id}/reveal          burn after reading: the text once, then 410
POST   /api/v1/pastes/{id}/reports         { "reason": "malware" }; 202, goes to moderators

The text is sent as the raw request body, not inside JSON. So the server can count bytes and stream them, without unpacking 10 MB of JSON. Anonymous creates get a delete token, shown once.

The server answers 413 if the paste is too big, 429 if someone creates too fast, 404 for an ID that never existed, and 410 Gone for an expired, deleted or burned paste. Legal takedowns get 451 (RFC 7725). The viewer page is one HTML page that fetches the raw text and colours it in the browser.

Step 2. Text in an object store

Where should the text live? Inside the database row is simplest: one write, one read, one transaction. But 24 TB of text would turn a 1.5 TB database into a 25 TB one. Backups and recovery would take far longer, for bytes we never search.

So the text goes into an object store, such as S3, one object per paste. It is the cheapest safe place to keep bytes. It costs about $0.023 per GB a month, and it already keeps copies in several data centres. A database on fast disks costs several times that, before copies and backups.

This creates a new problem. A create now writes to two systems, and there is no transaction that covers both. So the order is our guarantee: body first, row second. If we crash between the two, we leave a body that no row points at. Nobody can read it, and a cleanup job deletes it later. The other order could leave a row pointing at text that was never saved. That’s a broken paste.

Pastebin · architecture

Stage 2: Bodies in object storage

Color keyServiceDatabaseCacheQueue or streamBackground workerNew in this stageAsync or occasional

Stage

Paste views per second: 12,000

Stage 2 of 6

Bodies in object storage

Why this stage

Pastes go up to 10 MB and the bodies reach 24 TB, which do not belong in database rows.

At the 12,000 views/s peak, one server and one database are both far past capacity.

What we add

A load balancer, stateless app servers, and an object store holding one compressed object per paste.

Postgres keeps about 400 bytes of metadata per paste.

What it costs

A create is now two writes in two systems with no shared transaction, so a crash can leave an orphan blob.

At 12k paste views per second, Metadata DB is 120% busy. Stage 3 is the fix.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What one machine can do

One app server serves about 3,000 views a second of 20 KB pastes.
One Postgres node does about 10,000 point reads a second and 5,000 writes. While it also stores the paste bodies, it manages about half that.
One Redis node answers about 100,000 reads a second.
One scanner checks about 300 average pastes a second.

The object store

The object store spreads keys across partitions, each good for about 5,500 reads a second. Across our keys, that is about 200,000 a second.

What the caches save

The CDN answers about 85% of views, or 90% with an origin shield.
Redis answers about 95% of what is left.

PasteService.javapaste_service.pyPasteService.cspaste_service.gopaste-service.ts

// What the create path needs from the two stores, and nothing more.
interface BlobStore {
    void put(String key, byte[] bytes, byte[] sha256);  // the store checks the hash on arrival
}

interface MetadataStore {
    void insert(PasteRow row);  // a conditional insert: fails if the ID already exists
}

enum Visibility { PUBLIC, UNLISTED, PRIVATE }

record PasteRow(String id, String blobKey, int rawBytes, byte[] sha256,
                Visibility visibility, Instant expiresAt) {}  // expiresAt null: never

class TooLargeException extends RuntimeException {}

class PasteService {
    static final int MAX_BYTES = 10 * 1024 * 1024;  // also enforced at the load balancer
    static final String ALPHABET =
        "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ";

    private final BlobStore blobs;
    private final MetadataStore meta;
    private final Clock clock;
    private final SecureRandom random = new SecureRandom();  // unguessable, not just unique

    PasteService(BlobStore blobs, MetadataStore meta, Clock clock) {
        this.blobs = blobs;
        this.meta = meta;
        this.clock = clock;
    }

    String create(InputStream body, Visibility visibility, Duration ttl) throws IOException {
        byte[] text = readAtMost(body, MAX_BYTES);
        byte[] stored = gzip(text);              // served as stored: about a third of the bytes
        byte[] sha = sha256(stored);             // checked end to end
        String id = newId(visibility);
        String key = "p/" + id;                  // no random prefix needed: see the data model

        // The two writes cannot share a transaction, so their ORDER is the guarantee.
        // 1. Body first. A crash after this leaves an orphan blob nobody can reach,
        //    which the reconciler deletes later.
        blobs.put(key, stored, sha);
        // 2. Metadata second. Only now does the paste exist. The reverse order could leave
        //    a row pointing at a body that was never written: a paste that 500s forever.
        Instant expires = ttl == null ? null : clock.instant().plus(ttl);
        meta.insert(new PasteRow(id, key, text.length, sha, visibility, expires));
        return id;
    }

    // Unlisted and private IDs are the only secret protecting the paste, so they come from a
    // CSPRNG: 16 Base62 characters is about 95 bits. Public pastes could use the URL
    // shortener's short leased IDs instead; this sketch uses one scheme for all.
    String newId(Visibility visibility) {
        var sb = new StringBuilder(16);
        for (int i = 0; i < 16; i++) sb.append(ALPHABET.charAt(random.nextInt(62)));  // unbiased
        return sb.toString();
    }

    // Count bytes ourselves: Content-Length can lie and chunked uploads have none. Stop
    // reading at the limit so a 2 GB body costs us 10 MB of memory, not 2 GB.
    static byte[] readAtMost(InputStream in, int limit) throws IOException {
        byte[] bytes = in.readNBytes(limit + 1);
        if (bytes.length > limit) throw new TooLargeException();  // the API answers 413
        return bytes;
    }

    static byte[] sha256(byte[] bytes) {
        try {
            return MessageDigest.getInstance("SHA-256").digest(bytes);
        } catch (NoSuchAlgorithmException e) {
            throw new IllegalStateException(e);  // every JVM must ship SHA-256
        }
    }

    static byte[] gzip(byte[] text) throws IOException {
        var out = new ByteArrayOutputStream();
        try (var gz = new GZIPOutputStream(out)) {
            gz.write(text);
        }
        return out.toByteArray();
    }
}

A crash between the two writes leaves only an unreachable orphan

Scroll or pinch to zoom, drag to pan

Color key
Database
Service

The cleanup job compares the object store’s daily list of objects with the table. It deletes bodies older than a day that no row points at.

Step 3. A CDN and a metadata cache

At 12,000 views a second, the database would be 120% busy, and every view would download from our servers. But a paste never changes. Any copy is correct until the paste expires or is deleted.

So views go through a CDN, a network of caches close to users. It keeps public and unlisted pastes, but only until each paste expires. Behind it sits an origin shield: one regional cache that all CDN locations ask first. So many misses at the edge become one request to us. For views that do reach us, Redis holds the metadata, so the database is rarely asked.

Pastebin · architecture

Stage 3: A CDN and a metadata cache

Color keyServiceDatabaseCacheQueue or streamBackground workerNew in this stageAsync or occasional

Stage

Paste views per second: 12,000

Stage 3 of 6

A CDN and a metadata cache

Why this stage

At 12,000 views/s the metadata database is 120% busy, and every view is a full download from our origin.

A paste never changes, so any cached copy is correct until it expires.

What we add

A CDN that caches public and unlisted pastes with a TTL capped at their expiry, behind an origin shield, and a Redis cache of paste metadata.

What it costs

Deletes and takedowns need a CDN purge, and private and burn-after-read pastes always reach the origin.

At 12k paste views per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What one machine can do

One app server serves about 3,000 views a second of 20 KB pastes.
One Postgres node does about 10,000 point reads a second and 5,000 writes. While it also stores the paste bodies, it manages about half that.
One Redis node answers about 100,000 reads a second.
One scanner checks about 300 average pastes a second.

The object store

The object store spreads keys across partitions, each good for about 5,500 reads a second. Across our keys, that is about 200,000 a second.

What the caches save

The CDN answers about 85% of views, or 90% with an origin shield.
Redis answers about 95% of what is left.

The CDN answers about 85% of views, so our servers see about 1,800 a second at peak. Follow one view through the design. Next hop moves the request one step.

Pastebin · architecture

Trace a request: View a public paste, CDN miss

Color keyServiceDatabaseCacheQueue or streamBackground workerNew in this stageAsync or occasionalHop in motionHops doneResponse

View a public paste, CDN miss: step 1 of 6

Users to CDN edge

GET /raw/aZ3kP9qX at the nearest edge. The cache key is the URL. Most views end here in about 10 ms. This paste was evicted, so the edge asks the origin.

10 ms of a 150 ms budget

Users → CDN edge10 ms
CDN edge → Load balancer20 ms
Load balancer → App servers1 ms
App servers → Redis cluster1 ms
App servers → Object storage30 ms
Response

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

Private pastes and burn-after-reading pastes never go on the CDN. The second stress test explains the rules.

Step 4. Expiry and abuse, off the hot path

Two jobs must happen for every paste, but neither should slow down a create or a view.

Expiry. Every read already checks expires_at. So an expired paste is unreadable the second it expires, with no job running at all. An expiry sweeper only frees the bytes later, in batches.

Abuse. Each create puts a paste.created event on a queue. Abuse scanners read it and check the text for malware, spam and leaked passwords. A bad paste is taken down through the same API, which sets expires_at to now and clears the caches.

Pastebin · architecture

Stage 4: Expiry and abuse off the hot path

Color keyServiceDatabaseCacheQueue or streamBackground workerNew in this stageAsync or occasional

Stage

Paste views per second: 12,000

Stage 4 of 6

Expiry and abuse off the hot path

Why this stage

Requirements, not load: expired pastes must stop being served and their bytes deleted, and public pastes must be scanned for malware, spam and leaked secrets.

What we add

An event queue, a pool of abuse scanners that read new pastes and take down bad ones, and an expiry sweeper that deletes expired metadata and blobs in batches.

What it costs

Scanning is asynchronous, so a bad paste from a trusted creator is visible for up to a minute, and there are three more things to run.

At 12k paste views per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What one machine can do

One app server serves about 3,000 views a second of 20 KB pastes.
One Postgres node does about 10,000 point reads a second and 5,000 writes. While it also stores the paste bodies, it manages about half that.
One Redis node answers about 100,000 reads a second.
One scanner checks about 300 average pastes a second.

The object store

The object store spreads keys across partitions, each good for about 5,500 reads a second. Across our keys, that is about 200,000 a second.

What the caches save

The CDN answers about 85% of views, or 90% with an origin shield.
Redis answers about 95% of what is left.

This is enough for a mid-level answer. Text and details in separate stores, written in a safe order, a CDN with clear rules, and expiry checked on every read. Step 5 and the stress tests are what make it a senior answer.

Go deeperFollow a create through the design1 min

Pastebin · architecture

Trace a request: Create a paste

Color keyServiceDatabaseCacheQueue or streamBackground workerNew in this stageAsync or occasionalHop in motionHops doneResponse

Create a paste: step 1 of 6

Users to Load balancer

POST /api/v1/pastes with the text as the body and an Idempotency-Key header. The load balancer refuses bodies over 10 MB with 413 before they reach a server.

2 ms of a 300 ms budget

Users → Load balancer2 ms
Load balancer → App servers3 ms
App servers → Object storage40 ms
App servers → Metadata DB5 ms
App servers → Event queueasync
Response

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

If the event is lost after the row is saved, the row still says scan_status = pending. A small job sends the event again for any row pending more than 10 minutes.

Go deeperOther metadata databases that fit1 min

Option	Why it fits	What it costs	Pick it when
PostgreSQL	Fast lookups, “insert only if new”, a cheap scan by expiry	One machine for writes; every delete leaves work for cleanup	Below a few TB, and you want simple
DynamoDB	Key-value, scales writes, built-in expiry	Built-in expiry may take days to delete, so reads must still check	You’re on AWS and don’t want to run a database
Cassandra	Handles many writes, expiry per row, many regions	Expired rows leave markers that slow reads until cleanup	You already run Cassandra

Step 5. Split the metadata

At ten times the traffic, the CDN still handles reads. The database is the first limit: about 12,000 creates and deletes a second, against about 5,000, and a 15 TB table. The fix is the URL shortener’s: split metadata by a hash of the paste ID, so each lookup touches one shard. Each shard then runs its own sweeper. The object store splits itself, so it needs nothing from us.

Pastebin · architecture

Stage 5: Partition the metadata

Color keyServiceDatabaseCacheQueue or streamBackground workerNew in this stageAsync or occasional

Stage

Paste views per second: 12,000

Stage 5 of 6

Partition the metadata

Why this stage

At 10 times the traffic, 5,800 creates/s plus a later delete per paste is 240% of one primary, and the metadata heads for 15 TB.

What we add

Metadata split into 16 shards by hash of the paste ID, each with replicas and its own sweeper, and Redis as a cluster.

What it costs

A live migration, and "all pastes of an owner" needs its own table keyed by owner.

At 12k paste views per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What one machine can do

One app server serves about 3,000 views a second of 20 KB pastes.
One Postgres node does about 10,000 point reads a second and 5,000 writes. While it also stores the paste bodies, it manages about half that.
One Redis node answers about 100,000 reads a second.
One scanner checks about 300 average pastes a second.

The object store

The object store spreads keys across partitions, each good for about 5,500 reads a second. Across our keys, that is about 200,000 a second.

What the caches save

The CDN answers about 85% of views, or 90% with an origin shield.
Redis answers about 95% of what is left.

Go deeperSplitting without downtime1 min

The move follows the same steps as the URL shortener’s split. First, every lookup goes through a slot map that still points at the old database. Then new writes go to both places, old rows are copied slot by slot, and both copies are compared. Reads move a few slots at a time, and each move can be undone.

Pastes make one part easier and one part harder. Easier: a row never changes after it is created, except for expires_at, so a copied row stays correct. Harder: the sweeper deletes rows all the time. So during the move, each slot’s sweeper runs only on the side that owns the slot. Otherwise the two sides would delete the same rows twice, or miss some.

Stress it

The design is complete. Now we test it against the four things paste sites are known for: expiry, caching rules, a viral paste, and abuse. No new boxes appear here.

Expiring billions of pastes

Interviewerminute 20

Three million pastes expire every day. How do they actually go away?

What they are testing

Whether you separate “unreadable”, which must be exact, from “deleted”, which can be late.

There are two different promises here, and mixing them up causes trouble.

Unreadable at expiry is about correctness. A 10-minute paste must not be readable at minute 11.
Deleted eventually is about cost and privacy. It can be a little late.

If we treat them as one promise, we end up with a cleanup job that must run every second. So we split them. The check on every read handles the first promise, exactly and for free. A batched sweeper handles the second. It can be late without anyone noticing. On each shard, it takes the 1,000 oldest expired rows, deletes their bodies in one bulk request, then deletes the rows:

Sweeper.javasweeper.pySweeper.cssweeper.gosweeper.ts

record Expired(String id, String blobKey) {}

// What the sweeper needs from one metadata shard. Deletes, burns and takedowns all set
// expires_at to now, so this one query finds everything that must go.
interface ExpiredRows {
    // SELECT id, blob_key FROM pastes WHERE expires_at < $1 ORDER BY expires_at LIMIT $2,
    // a range scan on a partial index over expires_at.
    List<Expired> oldestExpired(Instant now, int limit);

    void deleteRows(List<String> ids);  // DELETE FROM pastes WHERE id = ANY($1)
}

interface BlobDeleter {
    // One DeleteObjects request takes up to 1,000 keys. Deleting a missing key succeeds,
    // which is what makes a retried batch harmless. Returns the keys that FAILED: the
    // request answers 200 even when some keys did, listing them in the body.
    Set<String> deleteMany(List<String> keys);
}

class Sweeper {
    static final int BATCH = 1_000;  // one bulk delete request, one short transaction

    private final ExpiredRows rows;
    private final BlobDeleter blobs;
    private final Clock clock;
    private final Runnable pause;  // throttle between batches; a no-op in tests

    Sweeper(ExpiredRows rows, BlobDeleter blobs, Clock clock, Runnable pause) {
        this.rows = rows;
        this.blobs = blobs;
        this.clock = clock;
        this.pause = pause;
    }

    // One pass over one shard. Readers never depend on this running on time: every read
    // checks expires_at, so a paste is gone the second it expires. This only frees bytes.
    int sweepOnce() {
        int total = 0;
        while (true) {
            List<Expired> batch = rows.oldestExpired(clock.instant(), BATCH);
            if (batch.isEmpty()) return total;
            // Blobs first. If we crash after this line, the rows are still there, still
            // expired, still answering 410, and the next pass finishes the job. Rows first
            // would orphan the blobs, with nothing left pointing at them.
            Set<String> failed = blobs.deleteMany(batch.stream().map(Expired::blobKey).toList());
            // Only rows whose blob is gone. Deleting the others would orphan their blobs.
            List<String> done = batch.stream()
                .filter(e -> !failed.contains(e.blobKey())).map(Expired::id).toList();
            rows.deleteRows(done);
            total += done.size();
            // Failures usually mean the store is throttling us (503 Slow Down): end the pass
            // and let the next one retry, rather than hammering it with the same keys.
            if (!failed.isEmpty() || batch.size() < BATCH) return total;
            // A backlog of millions (after an outage) must not flood the primary with
            // deletes and push replica lag up, so the sweeper paces itself.
            pause.run();
        }
    }
}

The order mirrors the create: bodies first, rows second. A crash after deleting the bodies leaves rows that still answer 410, and the next run finishes the job.

The numbers are small. 3 million expiring pastes a day is about 35 deletes a second. The danger is a backlog. After a day-long outage, 3 million rows are waiting. Even then, the sweeper clears them in under 20 minutes, if it slows down whenever the object store asks it to.

As a backup, pastes that expire are stored under a folder per expiry length (e/1d/, e/1w/). The object store’s own lifecycle rules delete whatever the sweeper missed, a day after the longest expiry in each folder.

Interviewer asksA user deletes a paste that contains their password. How fast is it really gone?

From our servers at once: expires_at is set to now, and the Redis entry is removed. From the CDN within a minute, after a purge. From browsers within 5 minutes. The bytes go at the sweeper’s next run. Copies someone already saved can’t be recalled. That’s why the form warns before publishing something that looks like a password.

Go deeperTwo details that keep the sweeper safe1 min

The bulk delete call in S3, DeleteObjects, answers 200 even when some keys failed. The failed keys are listed in the response body. So the sweeper deletes rows only for keys that really were deleted. Otherwise a row could vanish while its body lives on, unreachable but still stored.

Each shard’s sweeper also holds a lease, a lock that expires on its own, so only one copy runs at a time. If two overlap during a deploy, they just repeat the same deletes, which is harmless.

What the CDN may cache

Caching makes views cheap, but some pastes must never be cached. Which pastes may the CDN keep, and for how long?

A cached copy is safe when it can’t outlive the paste, and can’t reach the wrong reader. That gives one rule per kind of paste:

Paste	On the CDN	In browsers	Why
Public, never expires	A day, cleared on delete	5 minutes	The text never changes; browsers can’t be cleared
Public, expires	The time left, at most a day	Same, also capped	The CDN copy dies with the paste, with no purge
Unlisted	Same as public, and hidden from search	Same	The link is the secret, and the CDN stores by link
Private	Never	Never stored	The answer depends on who asks
Burn after reading	Never	Never stored	Exactly one reader

All of this is one small function that sets the headers on each response:

CachePolicy.javapaste.pyCachePolicy.cspaste.gopaste.ts

enum Visibility { PUBLIC, UNLISTED, PRIVATE }

record Paste(String id, Visibility visibility, boolean burnAfterReading, Instant expiresAt) {}

// The cache headers on every paste response. This is where "immutable" turns into a cache
// policy, and where the pastes that must never sit in a shared cache are kept out of one.
final class CachePolicy {
    static final long EDGE_MAX_SECONDS = 86_400;  // one day at the CDN; purges handle deletes
    static final long BROWSER_MAX_SECONDS = 300;  // browsers cannot be purged: keep it short
    static final long SKEW_SECONDS = 5;           // edge clocks and Age headers are not exact

    static Map<String, String> headers(Paste p, Instant now) {
        var h = new LinkedHashMap<String, String>();
        // A burn-after-reading paste has exactly one reader, and a private paste depends on
        // who is asking. Neither may be stored by a CDN or a corporate proxy.
        if (p.burnAfterReading() || p.visibility() == Visibility.PRIVATE) {
            h.put("Cache-Control", "private, no-store");
            return h;
        }
        long edge = EDGE_MAX_SECONDS;
        if (p.expiresAt() != null) {
            // An edge copy must die no later than the paste does. The caller has already
            // answered 410 for an expired paste; the margin may round this to zero.
            long left = Duration.between(now, p.expiresAt()).getSeconds() - SKEW_SECONDS;
            edge = Math.max(0, Math.min(edge, left));
        }
        long browser = Math.min(BROWSER_MAX_SECONDS, edge);
        // immutable: the bytes behind this URL never change, so a reload need not revalidate.
        h.put("Cache-Control", "public, max-age=" + browser + ", s-maxage=" + edge + ", immutable");
        // Unlisted is "anyone with the link", so caching by URL is safe. Keep it out of
        // search engines, which is the one way an unlisted link becomes listed.
        if (p.visibility() == Visibility.UNLISTED) h.put("X-Robots-Tag", "noindex");
        h.put("Surrogate-Key", "paste-" + p.id());  // one purge drops every variant, on CDNs that purge by tag
        return h;
    }
}

Private pastes get two locks. They are only served on a special signed path that the CDN is set up never to cache. And the response also says no-store. A bug in one of these alone can’t put a private paste in a shared cache.

Burn after reading is the one read that writes. Two people might open the link at the same moment. Only one may see the text. So the read is a conditional update on the main database: “mark it burned, but only if it isn’t burned yet”. The database lets exactly one of them win.

Two readers race for a burn-after-reading paste

Scroll or pinch to zoom, drag to pan

Color key
Database
Service

Go deeperDeletes, unlisted pastes, and the burn trace1 min

Deletes can’t wait for a cache to expire. So the API tells the CDN to purge the paste, through the queue, retrying until the CDN confirms. A lost purge would leave a deleted paste on the CDN for up to a day. Some CDNs purge worldwide in seconds, others take minutes. Our one-minute promise rules out the slow ones. Browsers can’t be purged at all, which is why they only keep pastes for 5 minutes.

Unlisted pastes can go on the CDN. Anyone with the link can read the paste. The CDN stores it by that same link. So a cached copy is exactly as reachable as the original. The real leaks are elsewhere. Search engines might index the page, so we tell them not to. And when a reader clicks a link inside the paste, the browser may tell that site where the click came from, so we turn that off too.

Follow a burn through the design:

Pastebin · architecture

Trace a request: Burn after reading

Color keyServiceDatabaseCacheQueue or streamBackground workerNew in this stageAsync or occasionalHop in motionHops doneResponse

Burn after reading: step 1 of 7

Users to CDN edge

The reader clicks Reveal, which sends a POST, so link preview bots that only GET the URL cannot burn it. The CDN never caches or collapses this path.

10 ms of a 150 ms budget

Users → CDN edge10 ms
CDN edge → Load balancer20 ms
Load balancer → App servers1 ms
App servers → Metadata DB5 ms
App servers → Object storage30 ms
App servers → Event queueasync
Response

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

A paste goes viral

Now a 1 MB paste of leaked source code gets 5,000 views a second, starting two minutes after it was created.

5,000 views a second of a 330 KB compressed paste is about 1.6 GB a second, for one paste. And one object in S3 can only serve about 5,500 reads a second.

The dangerous moment is the start, when no CDN location has the paste yet. Hundreds of them miss at once. Two features stop that from reaching us. Request collapsing makes each CDN location send one request, however many readers are waiting. And the origin shield turns 300 location misses into one.

A new paste goes viral: one origin request

Scroll or pinch to zoom, drag to pan

Color key
Database
Service
Cache

For the few views that still reach our servers, we use single flight, as in the URL shortener: on a cache miss, only one request per server fetches the paste, and the others wait for it.

Go deeperThe bill for a viral paste1 min

There is also a bill. A 1 MB paste viewed a million times is about 330 GB of traffic, roughly $15. That’s fine for a real viral paste. But it’s also how people abuse a paste site as free file hosting. So anonymous pastes get a bandwidth limit.

Abuse: malware, spam, secrets and size

Attackers use the site to host malware, and people keep pasting their own API keys. What do we build?

Paste sites attract attackers for the same reasons people like them. They are anonymous, free, reliable, and on a domain that firewalls allow. Attackers store malware there, which infected machines download by the raw link. People also leak passwords and keys, and spammers post links.

The question is how much to check before a paste becomes visible. Checking everything during the create is slow, and if a scanner breaks, nobody can create pastes. Checking everything afterwards leaves malware online for about a minute. So we check in three layers:

During the create, about 1 ms. Size, counted in bytes. Rate limits per account and IP address (the rate limiter). The text’s hash against a list of known bad files. And patterns for keys with fixed prefixes, such as AWS keys (AKIA...). A match on a public paste asks “make it private?” before publishing.
Hold the risky ones. A paste from an anonymous creator with a bad history is saved, but shown to nobody until its scan passes, usually within seconds.
In the scanners, seconds to a minute. Deeper malware rules, run after decoding hidden content. Checks for random-looking secrets. Spam detection. A hit takes the paste down, which sets expires_at to now, clears Redis and purges the CDN.

When things break

Interviewerminute 40

Your object store starts returning errors. What do users see?

What they are testing

Whether cached pastes keep loading, and nothing is lost, while other views and creates fail.

Views of existing pastes should survive almost anything. Three choices we already made help:

The CDN holds most views. If the object store or database has trouble, popular pastes keep loading.
The write order is safe. A create that fails half way leaves only an unreachable body, never a broken paste.
Scans and sweeps are queued. If they stop, they catch up later. Nothing breaks for users.

Click any box to break it and see what users would notice:

Pastebin · architecture

Break it: what fails, who notices, what users see

Color keyServiceDatabaseCacheQueue or streamBackground workerNew in this stageAsync or occasional

Break

Failure: click any box to break it

Object storage goes down

What breaks

Reads of uncached pastes and every create fail, or slow down sharply.

How we notice

Object store error rate and p99 latency per operation.

What users see

Pastes already on the CDN still load.

Uncached pastes and new creates fail.

What we do

Retries with backoff for slowdowns.

For a regional outage, fail reads to the region B copy.

Bytes are not lost: they are stored across several zones.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The object store keeps copies across the data centres of one region. But a whole region can go down. S3 in one US region was down for about four hours in 2017. Four hours is our whole yearly allowance. So we copy every body to a second region, keep a database copy there, and let the CDN switch to it. New pastes are still created only in the home region, so the two never conflict.

Pastebin · architecture

Stage 6: Survive a region

Color keyServiceDatabaseCacheQueue or streamBackground workerNew in this stageAsync or occasional

Stage

Paste views per second: 12,000

Stage 6 of 6

Survive a region

Why this stage

The object store is regional.

One regional outage of a few hours uses the whole 4.4 hour yearly budget of 99.95%, and durability promises do not cover losing a region.

What we add

Every blob replicated to a second region, a metadata replica there, and a CDN that fails over to it.

Creates stay in the home region.

What it costs

Twice the storage bill, and a paste younger than the replication lag may be missing after a failover.

At 12k paste views per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What one machine can do

One app server serves about 3,000 views a second of 20 KB pastes.
One Postgres node does about 10,000 point reads a second and 5,000 writes. While it also stores the paste bodies, it manages about half that.
One Redis node answers about 100,000 reads a second.
One scanner checks about 300 average pastes a second.

The object store

The object store spreads keys across partitions, each good for about 5,500 reads a second. Across our keys, that is about 200,000 a second.

What the caches save

The CDN answers about 85% of views, or 90% with an origin shield.
Redis answers about 95% of what is left.

The second region may be missing the last few minutes of pastes. A miss there returns “try again later” (503), not “gone” (404). And the copying doesn’t copy deletes, so the sweeper deletes in both regions.

Go deeperEvery failure, and what users see2 min

Part	Failure	What users see	How we notice	What we do
CDN	Outage	Slower views; our servers see every view	Our request rate, CDN errors	Servers sized for 3 times their share; a second CDN
App server	Crash	A few failed requests	Load balancer health checks	Replaced; a half-done create leaves only an orphan body
Redis	Node down	Views that miss the CDN get a few ms slower	Hit rate drops	Backup takes over; single flight per paste
Metadata primary	Down	Creates, deletes and burns fail; views continue	Write errors	Failover in about 30 s; clients retry with their key
Object store	Errors	Uncached views and creates fail; nothing lost	Errors and slow responses	Retry with backoff; read from the second region
Queue	Down	Scans and CDN purges wait	Send errors, queue lag	Rows stay pending; a job sends them again
Scanners	Behind	Bad pastes stay up longer	Age of the oldest unscanned paste	Add scanners; hold every new anonymous paste until scanned
Sweeper	Stopped	Nothing; storage grows	Age of the oldest expired row	Restart; batches catch up; lifecycle rules as backup
Whole region	Down	Views move to region B; creates pause	Test requests from outside	CDN switches; promote region B for writes if it lasts

When overloaded, we keep views first, then creates, then scans, then sweeps.

Our own code is a likelier way to lose data than disks. A buggy sweeper could delete live pastes. So the object store keeps old versions for 7 days, and the database can be restored to any moment.

Go deeperWhat each action promises1 min

Action	Promise	Why it is enough
Create	Body and row both saved before “created”	The order means a crash leaves an orphan, never a broken paste
Creator’s first view	At once	Metadata goes into Redis on create; S3 reads its own writes
Everyone else’s view	Within a second, never wrong	A paste never changes; a copy can be “not there yet”, never “wrong”
Expiry	Exact on our servers; the CDN within seconds	Reads check the clock; CDN lifetimes stop at the expiry time
Delete and takedown	Us at once, CDN within a minute, browsers in 5	Retried purges on a fast CDN, and short browser lifetimes
Burn after reading	Exactly one reader	A conditional update on the main database, never a cache or a copy
Abuse scan	Within minutes	Risky pastes are held until scanned

Some people still say “object storage is eventually consistent, so wait before reading”. That was true of S3 years ago. Since December 2020, S3 always returns what you just wrote (AWS).

Go deeperRunning it: alerts and cost1 min

Wake someone up when: fewer than 99.9% of views work for 5 minutes; the slowest 1% of views at our servers take over 300 ms; the CDN hit rate drops below 80%, which predicts overload and a bigger bill; the oldest unscanned public paste is over 10 minutes old, because that’s malware staying up; or the oldest expired row is over an hour old.

Cost. The bill is small, but its shape matters. Storage grows with live pastes, upload requests with creates, and traffic with views. Two traps: a 2 KB paste costs more to upload than to store for ten years. And cheaper “infrequent access” storage bills every object as at least 128 KB and 30 days, so moving small or short-lived pastes there raises the cost.

Wrap up

The whole design

Here is everything we built, in the order we built it. Use the buttons to step through it.

Pastebin · architecture

The full design, stage by stage

Color keyServiceDatabaseCacheQueue or streamBackground workerNew in this stageAsync or occasional

Stage

Paste views per second: 12,000

Stage 6 of 6

Survive a region

Why this stage

The object store is regional.

One regional outage of a few hours uses the whole 4.4 hour yearly budget of 99.95%, and durability promises do not cover losing a region.

What we add

Every blob replicated to a second region, a metadata replica there, and a CDN that fails over to it.

Creates stay in the home region.

What it costs

Twice the storage bill, and a paste younger than the replication lag may be missing after a failover.

At 12k paste views per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.The whole design. Step through it: one table, text in object storage, a CDN, expiry and abuse jobs, shards, then a second region.

The numbers behind the busy bars

What one machine can do

One app server serves about 3,000 views a second of 20 KB pastes.
One Postgres node does about 10,000 point reads a second and 5,000 writes. While it also stores the paste bodies, it manages about half that.
One Redis node answers about 100,000 reads a second.
One scanner checks about 300 average pastes a second.

The object store

The object store spreads keys across partitions, each good for about 5,500 reads a second. Across our keys, that is about 200,000 a second.

What the caches save

The CDN answers about 85% of views, or 90% with an origin shield.
Redis answers about 95% of what is left.

The decisions, and what each one costs

Decision	We chose	We gave up	We’d choose differently if
Where text lives	Object store, one object per paste	One write and one transaction per create	Pastes were capped at 64 KB (all in the database)
IDs	Short for public, 16 random characters otherwise	One ID scheme	There were no unlisted pastes
Metadata store	Postgres, split by ID later	Built-in expiry and easy write scaling	We were on AWS and wanted no database (DynamoDB)
Expiry	Checked on read, swept in batches, lifecycle backup	Instant physical deletion	The law required erasure within minutes
Caching	CDN up to expiry, short browser lifetimes	Instant deletes everywhere	Pastes were mostly private (no CDN)
Abuse	Cheap checks first, deep scans after, hold the risky	Zero exposure for trusted creators	Accounts were required
Regions	Bodies copied, creates in a home region	Half the storage bill; creates during an outage	99.9% were enough

How to spend 45 minutes

0 to 5. Size, expiry, who can read, and burn after reading.
5 to 10. 58 creates and 12,000 peak views a second; 24 TB of text against 1.5 TB of details; requests and traffic cost more than storage.
10 to 20. Build it: one table, then text in object storage with a safe write order, the CDN, and expiry on read. A mid-level answer can stop here.
20 to 31. The stress tests: expiry and the caching rules, including burn after reading.
31 to 40. The viral paste or abuse, whichever the interviewer picks.
40 to 45. The object store failing, then your own questions.

Never skip the write order, expiry on read, or which pastes the CDN may keep.

Follow-up questions

Interviewer asksTwo people paste the same 5 MB log file. Should you store it once?

Storing by hash would, but then deletes need a count of who uses each body. A race between “last user deleted” and “new user added” could delete a body someone just pasted. It would also let anyone test whether a text was ever pasted. Identical large pastes are rare, so each paste gets its own object.

Interviewer asksWhy not let clients upload large pastes straight to the object store?

That’s right for file sharing. For text, the server must see the body anyway: to count bytes, check it’s valid text, look for leaked keys and compress it.

Go deeperMore follow-up questions2 min

Half of your pastes are under 2 KB. Is one object per paste wasteful? In upload requests, yes. If that cost dominates, store bodies under about 4 KB inside the database row. That adds a second read path and a few TB of database, so do it only when the bill says so.

A paste set to expire in 10 minutes is still on a CDN location at minute 11. How? Clocks that disagree by a few seconds, or a shield that served it with an age the location ignored. A test that creates a 10-minute paste and checks it from several regions at minute 10 and a half catches this.

Someone is using your raw links to control a botnet. What do you see, and do? One small paste fetched by many IP addresses on a regular schedule, often by programs rather than browsers. Spot it in the CDN logs per paste, take it down, and keep takedowns fast. Attackers paste again within minutes.

Product wants a public feed of the newest pastes. What changes? Don’t query every shard. Keep a separate read model: listen for pastes that passed scanning, keep the newest few thousand in a short list, and serve it with a short CDN lifetime.

Design Pastebin

The answer in one minute

Scope it

The numbers that decide it

Build the design

Step 1. One server, one table

Stage 1: One server, one table

Why this stage

What we add

What it costs

Step 2. Text in an object store

Stage 2: Bodies in object storage

Why this stage

What we add

What it costs

Step 3. A CDN and a metadata cache

Stage 3: A CDN and a metadata cache

Why this stage

What we add

What it costs

Trace a request: View a public paste, CDN miss

Step 4. Expiry and abuse, off the hot path

Stage 4: Expiry and abuse off the hot path

Why this stage

What we add

What it costs

Trace a request: Create a paste

Step 5. Split the metadata

Stage 5: Partition the metadata

Why this stage

What we add

What it costs

Stress it

Expiring billions of pastes

What the CDN may cache

Trace a request: Burn after reading

A paste goes viral

Abuse: malware, spam, secrets and size

When things break

Break it: what fails, who notices, what users see

What breaks

How we notice

What users see

What we do

Stage 6: Survive a region

Why this stage

What we add

What it costs

Wrap up

The whole design

The full design, stage by stage

Why this stage

What we add

What it costs

The decisions, and what each one costs

How to spend 45 minutes

Follow-up questions

Further reading

Keep reading

The designbeyondcode digest, every Saturday