Blob storage

The problem it solves

A blob is a lump of bytes the system stores and returns but never looks inside: a photo, a video, a backup, a log file. A photo app takes 1 million uploads a day at 2 MB each: 2 TB a day, about 730 TB after a year. The first version stores each photo in a BYTEA column next to its caption. It works at a few thousand photos, and then four things go wrong at once.

1. Cost. Database storage is provisioned SSD at about $0.08 per GB-month, held three times (a primary and two replicas), plus backups. 730 TB costs about $175,000 a month. The same bytes in S3 Standard cost about $17,000, with copies in three availability zones included.
2. Size. No single database server holds 730 TB. Managed Postgres and MySQL top out at tens of TB per instance. So the photos would force sharding, on a table with no other reason for it.
3. Operations. Backups, restores and replica rebuilds scale with bytes, from minutes to days.
4. The serving path. Every view drags bytes through the buffer pool and an app server. And a 2 MB upload over a 1 Mbit/s mobile link holds an app server connection for 16 seconds.

The fix splits the data by shape. The metadata is small, structured and queried: owner, caption, size, and which key holds the bytes. So it stays in the database, at about 1 KB a photo, or 365 GB a year. The bytes go to an object store. It’s built for exactly one thing: put a blob under a name, get it back by that name, and never lose it.

How it works

Step 1: Files on a server’s disk

The simplest version writes each upload to /var/data/photos/{id}.jpg, and stores the path in the row. It fails at the second server, or the first dead disk, because the file lives on one machine. A shared network filesystem moves the problem to one big server. Its POSIX semantics (locks, rename, partial writes) are expensive to scale, and photos never needed them.

Step 2: An object store, and what an object is

An object store drops everything a filesystem promises except the parts blobs need. The model has three nouns:

• A bucket is a namespace with its own settings: region, access policy, versioning, lifecycle rules, encryption. An account has few buckets and each holds unlimited objects.
• A key is the object’s full name, such as media/u42/2026/10/7f3a.jpg. The namespace is flat: the slashes are just characters. “Folders” in a console are a display trick, a LIST with a prefix and a / delimiter.
• An object is the bytes plus a little metadata (size, content type, an ETag, user-defined headers). It is immutable: you replace a whole object, you never edit byte 1,000 of it.

The API is about six verbs: PUT, GET (optionally a byte range), HEAD, DELETE, LIST and COPY. The constraints that follow show up in every design:

Constraint	What it means	Design consequence
No partial update	Changing one byte rewrites the object	Split big mutable things into chunks (file sync), or append as new objects (logs, segments)
No rename	A “rename” is a copy plus a delete; renaming a folder of 1 million objects is 2 million requests	Never encode mutable things (titles, owners, folder paths) in keys; the database holds the name
LIST is slow and paged	1,000 keys per call, in key order; listing 1 billion keys is 1 million sequential calls	The database is the index. For audits, use a daily inventory report, not LIST
No transactions across objects	Two PUTs are two independent writes	A multi-object change needs a manifest object or a database commit as the single switch (step 4)
Per-request cost and latency	About $0.005 per 1,000 PUTs, tens of ms to the first byte	Tiny objects are expensive per byte; pack them (step 8)
Throughput per key prefix	About 3,500 writes and 5,500 reads a second per partition of the key space, which S3 splits as load grows	Spread hot write paths across prefixes; never serve a viral object straight from the bucket

Consistency. Since December 2020, S3 gives strong read-after-write consistency for every PUT, overwrite and DELETE. And LIST reflects them immediately. Google Cloud Storage and Azure Blob Storage are also strongly consistent for single-object operations.

Nothing is guaranteed across objects or copies. Two writers to one key race, and the last one wins. (S3’s conditional writes, If-None-Match and If-Match, added in 2024, prevent that.) Cross-region replication is asynchronous. And a CDN serves whatever it cached.

Common mistake

S3 is eventually consistent, so read your own writes from somewhere else

That was true until December 2020, when S3 overwrites and deletes could briefly return the old version and listings could miss new keys. Many courses still teach it. Today a GET after a successful PUT returns the new object every time. The consistency problems that remain are between the store and your database, between regions, and between the store and the CDN.

Step 3: Durability, from copies to erasure codes

S3 Standard is designed for 99.999999999% durability, eleven nines, per object per year. AWS’s own illustration: store 10 million objects, and expect to lose one every 10,000 years. Availability is a different, much lower number. S3 Standard is designed for 99.99%, which allows about 53 minutes of errors a year. Durability means “is it still there”. Availability means “can I read it right now”.

There are two ways to buy durability from disks that fail at 1 to 2% a year.

Replication stores whole copies on different disks, racks or zones. Three copies survive any two losses and cost 3 times the data. Reads are simple, and repairing a lost copy reads one other copy.

Erasure coding cuts an object into k data fragments, and computes m parity fragments. Any k of the k + m fragments rebuild the object. Reed-Solomon RS(10,4) stores 10 data and 4 parity fragments, on 14 different disks. It survives any 4 losses, at 1.4 times the data.

That is more loss tolerance than three copies, for less than half the disk. That’s why every large object store codes most of its bytes. The price is repair and read cost. Rebuilding one lost fragment reads 10 others. And a read during a failure must reconstruct.

RS(10,4): any 10 of 14 fragments rebuild the object

Scroll or pinch to zoom, drag to pan

• Color key
• Service
• Database
• Failure

Scheme	Storage overhead	Losses survived	Repair of one fragment reads	Used by
3 replicas	3.0×	2	1 copy	Hot data, small objects, HDFS default, most databases
RS(6,3)	1.5×	3	6 fragments	HDFS erasure coding policy
RS(10,4)	1.4×	4	10 fragments	Facebook’s f4 warm photo store, many object stores
LRC(12,2,2)	1.33×	Any 3, most sets of 4	6 fragments	Azure Storage: local parities make the common single repair cheap

The overhead numbers are why erasure coding matters in a design. At 1,000 PB, the difference between 3 copies and 1.4 is 1,600 PB of disk. The Instagram problem uses exactly that move for media older than 90 days.

Go deeperThe durability arithmetic, and why it is optimistic3 min

Data is lost when more fragments fail than the code tolerates, before repair finishes. Take independent disks with a 2% annual failure rate. The chance that one given disk fails in a one-day repair window is 0.02 / 365, about 5.5 × 10⁻⁵. Then, per group of fragments, per year:

Scheme	Repair in 1 day	Repair in 7 days
3 copies	about 2 × 10⁻¹⁰ (10 nines)	about 9 × 10⁻⁹ (8 nines)
RS(6,3)	about 2 × 10⁻¹² (12 nines)	about 6 × 10⁻¹⁰ (9 nines)
RS(10,4)	about 2 × 10⁻¹⁵ (15 nines)	about 4 × 10⁻¹² (11 nines)

Two lessons follow.

First, repair speed matters as much as the code. The loss probability scales with the repair window, raised to the number of failures tolerated. So a store repairs in parallel from many disks, to keep the window in hours.

Second, these numbers assume independence, and real failures are not independent. A rack loses power. A firmware bug hits one disk model. A deploy corrupts a batch. An operator deletes the wrong thing. Correlated failures dominate real loss. So stores spread fragments across racks and availability zones. They scrub (re-read and checksum) every fragment all the time, to find silent corruption. And “eleven nines” is a design target, not a measurement. The biggest remaining risk to your data is your own code and credentials (“How it fails”), not disks.

Step 4: Metadata in a database, bytes in the store

Every real design has two stores. One is a database row per blob, which the application queries. The other is the object store: a dumb, huge, durable map from key to bytes.

The standard shape: rows in the database, bytes in the object store, reads at the edge

Scroll or pinch to zoom, drag to pan

• Color key
• Service
• Database

The two stores share no transaction, so a crash can always land between them, leaving one of two inconsistencies:

• A row that points at nothing. Readers get a broken image, and nothing looks wrong from the row, so nothing finds it. This is the bad one.
• Bytes no row points at, an orphan. Invisible to users, costs money, and a background job can find it by comparing the bucket’s inventory with the table.

So the rule is bytes first, metadata last. The commit that makes a blob visible is the last write. So a crash before it leaves only an orphan.

The production version adds a pending row:

1. Insert the row first, in state uploading. Readers can’t see it. It records the key and the upload id.
2. Upload the bytes.
3. Flip the row to ready.

The row now says what was in flight. So a sweeper can finish or clean up a stuck upload, without listing the bucket.

Keys follow from this. Use a fresh, opaque key per upload. Never use the user’s filename, and never overwrite in place.

The other half of the rule is a reconciler. It joins the daily inventory with the table. It deletes orphans older than a grace period. It rolls forward, or fails, rows stuck in uploading. It aborts multipart uploads nobody tracks. And it refuses to run when the numbers look wrong, because a truncated table makes every object look like an orphan.

JReconciler.javaPyupload.pyC#Reconciler.csGoupload.goTSupload.ts

JavaPythonC#GoTypeScript

JavaPythonC#GoTypeScript

import java.util.ArrayList;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;

// One line of the daily inventory listing, not a LIST call per key.
record StoredObject(String key, long size, int ageHours) {}

// The metadata database; only "ready" rows are visible to readers.
class Row {
    final int id;
    final String key;
    String state; // "uploading", "ready" or "failed"
    final long size; // what the client said it would upload
    final String uploadId;
    final int ageHours;

    Row(int id, String key, String state, long size, String uploadId, int ageHours) {
        this.id = id;
        this.key = key;
        this.state = state;
        this.size = size;
        this.uploadId = uploadId;
        this.ageHours = ageHours;
    }
}

// An unfinished multipart upload: invisible, but its parts are billed.
record Upload(String key, String uploadId, int ageHours) {}

class Reconciler {
    static final int GRACE_HOURS = 24; // younger objects may belong to an upload about to commit
    static final double MAX_DELETE_SHARE = 0.01; // over 1% of the bucket in one run means a bug

    // Read the listing before the rows. Every upload writes a fresh key, so an object that is
    // old in the listing can only gain a row by an upload that is already old too. Content
    // addressed keys break this: a new commit can reuse an old object, and needs a touch guard.
    static List<String> reconcile(List<StoredObject> objects, List<Row> rows, List<Upload> uploads) {
        List<String> actions = new ArrayList<>();
        Map<String, StoredObject> byKey = new HashMap<>();
        for (StoredObject o : objects) byKey.put(o.key(), o);
        Set<String> referenced = new HashSet<>();
        Set<String> tracked = new HashSet<>();
        for (Row r : rows) {
            referenced.add(r.key);
            if (r.uploadId != null) tracked.add(r.uploadId);
        }

        // 1. Orphans: bytes no row points at, typically a crash between the upload and the commit.
        List<StoredObject> orphans = new ArrayList<>();
        for (StoredObject o : objects) {
            if (!referenced.contains(o.key()) && o.ageHours() > GRACE_HOURS) orphans.add(o);
        }
        if (orphans.size() > Math.max(10, MAX_DELETE_SHARE * objects.size())) {
            // A truncated table or a bad join would make every object look orphaned. Stop and page.
            throw new IllegalStateException(orphans.size() + " orphans is too many; refusing to delete");
        }
        for (StoredObject o : orphans) actions.add("delete orphan " + o.key());

        // 2. Stuck uploads: the row says uploading and nothing has happened for a day.
        for (Row r : rows) {
            if (!r.state.equals("uploading") || r.ageHours <= GRACE_HOURS) continue;
            StoredObject obj = byKey.get(r.key);
            if (obj != null && obj.size() == r.size) {
                // The bytes made it; the crash was before the flip. Rolling forward keeps the
                // user's upload. (Production also checks the checksum the client declared.)
                r.state = "ready";
                actions.add("roll forward media " + r.id);
            } else {
                r.state = "failed";
                actions.add("abort upload " + r.uploadId + ", fail media " + r.id);
            }
        }

        // 3. Multipart uploads with no row at all. A lifecycle rule (AbortIncompleteMultipartUpload)
        // does the same after N days; this is the safety net for buckets without one.
        for (Upload u : uploads) {
            if (!tracked.contains(u.uploadId()) && u.ageHours() > GRACE_HOURS) {
                actions.add("abort upload " + u.uploadId() + " (no row)");
            }
        }
        return actions;
    }
}

class Main {
    public static void main(String[] args) {
        List<StoredObject> objects = List.of(
                new StoredObject("m/1", 2_000_000, 400), // a normal, committed photo
                new StoredObject("m/2", 3_000_000, 30), // orphan: the server crashed before the insert
                new StoredObject("m/3", 1_000_000, 2), // no row yet, but young: maybe committing now
                new StoredObject("m/4", 5_000_000, 26)); // complete, but the row never flipped to ready
        List<Row> rows = List.of(
                new Row(1, "m/1", "ready", 2_000_000, null, 400),
                new Row(4, "m/4", "uploading", 5_000_000, "up-4", 26),
                new Row(5, "m/5", "uploading", 9_000_000, "up-5", 30), // the phone gave up halfway
                new Row(6, "m/6", "uploading", 4_000_000, "up-6", 1)); // still in progress
        List<Upload> uploads = List.of(
                new Upload("m/5", "up-5", 30), new Upload("m/6", "up-6", 1), new Upload("m/9", "up-9", 50));
        for (String action : Reconciler.reconcile(objects, rows, uploads)) System.out.println(action);
        // delete orphan m/2
        // roll forward media 4
        // abort upload up-5, fail media 5
        // abort upload up-9 (no row)
    }
}

TradeoffBytes first, metadata last, with a reconciler

We gain

No reader ever sees a row without its bytes, with no distributed transaction, and every crash leaves something a background job can find and fix.

We give up

Orphans exist until the reconciler runs, and a grace period delays cleanup by a day. The reconciler can delete live data if it reads a broken table, so it needs a circuit breaker.

Step 5: Bytes skip your servers: presigned URLs

The bytes need not pass through your API at all. A presigned URL is a URL to one object, for one method, until one expiry time, signed with your credentials. The API checks that the user may upload, creates the row, signs a URL and returns it. The client then PUTs straight to the store, and the store checks the signature. App servers handle a 1 KB request instead of a 2 MB stream, and upload bandwidth becomes the store’s problem.

A presigned upload: the API signs, the store receives

Scroll or pinch to zoom, drag to pan

• Color key
• Database
• Service

A signature binds exactly what was signed, and the holder chooses everything else, so what you sign is the security design:

• Method and key, always. A PUT URL cannot read, and cannot write anywhere else.
• Content type, so a URL for image/jpeg cannot upload text/html that your domain would then serve to other users.
• A checksum (S3 accepts x-amz-checksum-sha256), so the bytes that arrive are the bytes the client declared.
• Size. A presigned PUT cannot express a maximum: it can at most pin one exact Content-Length if you sign that header. S3’s presigned POST policy can set a range (content-length-range). Either way, check the size with a HEAD before marking the row ready, and delete what is too big.
• Expiry, in minutes, not days. S3 caps presigned URLs at 7 days, and a URL signed with temporary credentials dies when they do. A leaked URL is a capability until it expires.

A presigned GET, valid for minutes, serves private downloads the same way. The code shows the idea in simplified form. S3’s real scheme, SigV4, adds a canonical request format and derived keys.

JPresigner.javaPyexample.pyC#Presigner.csGomain.goTSexample.ts

JavaPythonC#GoTypeScript

JavaPythonC#GoTypeScript

import java.net.URI;
import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;
import java.util.HashMap;
import java.util.HexFormat;
import java.util.Map;
import java.util.regex.Pattern;
import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;

class Presigner {
    // Never leaves the server. The client only ever sees signatures made with it. (S3's SigV4
    // goes further and derives a key per day, region and service, so a leaked key has a short life.)
    private static final byte[] SECRET = "server-side-secret".getBytes(StandardCharsets.UTF_8);
    static final long MAX_TTL = 7 * 24 * 3600; // S3 refuses presigned URLs longer than 7 days
    private static final Pattern SAFE_KEY = Pattern.compile("[a-z0-9/._-]+"); // server-picked keys stay URL-safe

    // Everything the server wants to enforce goes into the signed string. Whatever is left out,
    // the holder of the URL chooses freely.
    private static String canonical(String method, String path, long expires, String contentType) {
        return method + "\n" + path + "\n" + expires + "\n" + contentType;
    }

    private static String signature(String method, String path, long expires, String contentType) {
        try {
            Mac mac = Mac.getInstance("HmacSHA256");
            mac.init(new SecretKeySpec(SECRET, "HmacSHA256"));
            byte[] raw = mac.doFinal(canonical(method, path, expires, contentType).getBytes(StandardCharsets.UTF_8));
            return HexFormat.of().formatHex(raw);
        } catch (Exception e) {
            throw new IllegalStateException(e);
        }
    }

    static String sign(String method, String bucket, String key, long now, long ttl, String contentType) {
        if (ttl > MAX_TTL || !SAFE_KEY.matcher(key).matches()) {
            throw new IllegalArgumentException("refusing to sign");
        }
        String path = "/" + bucket + "/" + key;
        long expires = now + ttl;
        return "https://blob.example.com" + path + "?expires=" + expires + "&sig="
                + signature(method, path, expires, contentType);
    }

    // Runs on every request at the storage side. It needs no database: the URL carries
    // everything, which is why one signer can hand out millions of URLs a second.
    static boolean verify(String method, String url, String contentType, long now) {
        URI uri = URI.create(url);
        Map<String, String> query = new HashMap<>();
        if (uri.getRawQuery() != null) {
            for (String pair : uri.getRawQuery().split("&")) {
                String[] kv = pair.split("=", 2);
                if (kv.length == 2) query.put(kv[0], kv[1]);
            }
        }
        long expires;
        try {
            expires = Long.parseLong(query.getOrDefault("expires", ""));
        } catch (NumberFormatException e) {
            return false;
        }
        String sig = query.get("sig");
        if (sig == null || now >= expires) {
            return false; // an expired URL is dead, whatever else it says
        }
        String want = signature(method, uri.getRawPath(), expires, contentType);
        // Constant-time compare, so response timing does not leak the signature byte by byte.
        return MessageDigest.isEqual(want.getBytes(StandardCharsets.UTF_8), sig.getBytes(StandardCharsets.UTF_8));
    }
}

class Main {
    public static void main(String[] args) {
        long now = 1_700_000_000L;
        String url = Presigner.sign("PUT", "media", "uploads/u42/7f3a.jpg", now, 900, "image/jpeg");
        System.out.println(url);
        System.out.println(Presigner.verify("PUT", url, "image/jpeg", now + 60)); // true
        System.out.println(Presigner.verify("GET", url, "image/jpeg", now + 60)); // false: signed for PUT only
        System.out.println(Presigner.verify("PUT", url.replace("u42", "u43"), "image/jpeg", now + 60)); // false: key is signed
        System.out.println(Presigner.verify("PUT", url, "text/html", now + 60)); // false: no HTML served from our domain
        System.out.println(Presigner.verify("PUT", url, "image/jpeg", now + 901)); // false: expired after 15 minutes
    }
}

TradeoffDirect uploads with presigned URLs

We gain

Upload bandwidth and slow clients never touch the app fleet, which sizes for request rate alone, and the store’s own scaling handles the bytes.

We give up

You cannot inspect bytes in flight. So validation (type sniffing, size, malware scanning) moves to an asynchronous step after the upload. And a two-call flow (sign, then complete) needs cleanup for clients that never call complete.

Step 6: Big files, multipart and resumable uploads

A single PUT is all or nothing. Say a phone uploading a 2 GB video loses signal at 1.9 GB. It starts again from zero. And S3 refuses single PUTs over 5 GB anyway. Multipart upload splits the object into parts:

1. CreateMultipartUpload returns an upload id.
2. The client PUTs the parts. Each is 5 MB to 5 GB (only the last may be smaller), with up to 10,000 parts. It sends several at a time, in any order. Each part gets its own presigned URL, and each returns an ETag.
3. CompleteMultipartUpload, with the list of part numbers and ETags, stitches them into one object atomically. Before it, GET returns 404. After it, GET returns the whole object.
4. If the connection drops, ListParts tells the client which parts the store already has, and it sends only the rest.

Part size is a real decision. Smaller parts lose less on a dropped connection. Bigger parts mean fewer requests. The 10,000 part limit sets a floor: a 100 GB file needs parts of at least 10 MB. Mobile clients commonly use 5 to 16 MB parts, and servers 64 to 128 MB. Parallel parts also lift throughput, because one TCP connection to S3 tops out around 50 to 100 MB/s.

Try it below. Start the upload, drop the connection after a few parts, then resume. Watch only the missing parts go. Switch to Row first to see what viewers get meanwhile. Turn on the crash to compare what each order leaves behind. Then skip ahead 7 days, to see what the lifecycle rule and the reconciler clean up, and what they cannot.

Write order

Row firstBytes firstPending row

Start uploadDrop connectionResumeSkip ahead 7 daysReset Server crashes before its last write

Phone

12345678

64 MB video, 8 parts of 8 MB, 2 at a time

On the wire: 0 MB

Object store

Multipart upload

none

Object m/42.mp4

does not exist

Billed storage

0 MB

Metadata row

media 42

no row

object_key

-

upload_id

-

A viewer opens media 42:404, no such media

Pick a write order, then start the upload.

Two details matter in production. First, unfinished multipart uploads are invisible, and billed forever. So every bucket that accepts them needs the lifecycle rule AbortIncompleteMultipartUpload (7 days is common). Second, a multipart object’s ETag is not the MD5 of its bytes. It’s a hash of the part hashes, with -N appended. So integrity checks need an explicit checksum.

Go deeperResumable sessions instead of multipart1 min

Multipart is resumable because the client keeps the state: the upload id, and the parts done. Other protocols put the state on the server. A Google Cloud Storage resumable upload is one session URL. It accepts chunks, and answers “how many bytes do you have” with a byte offset. tus is an open protocol for the same thing, over plain HTTP. Either way, the client can lose its connection, its app process or its battery, and continue from the last confirmed byte.

Step 7: Reads: byte ranges and a CDN

An object store answers a GET in tens of milliseconds to the first byte. It supports byte-range GETs (Range: bytes=0-8388607). So a downloader can fetch 8 to 16 MB ranges in parallel. And a Parquet reader can fetch only the footer and the columns it needs.

For anything public or popular, put a CDN in front. It caches objects 10 to 40 ms from users. And S3 to CloudFront transfer is free, while S3’s internet egress starts around $0.09 per GB. Three practices make it work:

• Immutable keys, cached forever with Cache-Control: public, max-age=31536000, immutable. A new version is a new key, so nothing ever needs purging.
• The bucket stays private. Only the CDN’s own identity can read it (CloudFront’s origin access control, for example). Private content uses short-lived signed CDN URLs or cookies.
• An origin shield, a regional cache tier that turns a miss at 200 edges into one origin read.

Step 8: Content addressing and dedup

So far keys are random IDs. A different choice names each object by the hash of its content, such as its SHA-256. Git objects, container image layers (sha256:... digests), backup tools and the file sync problem all do this.

The properties fall out of the name. The same content always gets the same key, so a second upload of an identical file is a no-op: dedup for free. The key is a checksum, so a reader can verify what it got. And content at a key can never change, so it caches forever. File sync goes further and hashes chunks of about a megabyte, so editing one cell of a 200 MB spreadsheet uploads one chunk.

What it costs: deletion becomes reference counting, since two users can point at one object. So a mark-and-sweep collector must handle a race, where a new upload reuses an object it is about to delete.

And dedup leaks. Say “I already have this hash” skips the upload. Then a hash becomes proof that you have someone else’s file. Only skip uploads of blocks the same user already holds. And verify the hash of what arrives, instead of trusting the client’s.

Small objects need one more trick. A billion 1 KB thumbnails is 1 TB. It costs $23 a month to store, and $5,000 in PUT requests to write. Large systems pack small blobs into big segment files, and store (segment, offset, length) in the metadata. Facebook’s Haystack did this for photos, and web crawlers do it with 1 GB WARC files.

Options and tradeoffs

Where the bytes live

Option	Wins when	Loses
In the database row	Blobs are small (under about 100 KB), few, and must commit atomically with the row	Cost per GB, backup and failover times, buffer pool pollution
Network filesystem (NFS, cloud files)	Legacy software that needs POSIX paths, shared scratch space	Scaling, cost, semantics nobody needed
Object store, one key per blob	Almost everything: media, documents, backups, logs	Two stores to keep consistent, per-request costs, tens of ms latency
Object store, packed segments	Billions of tiny objects, append-heavy data (crawls, log chunks)	Deletes need compaction, you build an index and a small storage engine

An inline threshold is a reasonable refinement. A pastebin can keep bodies under a few KB in the row, and put the rest in the store. The price is two read and delete paths.

TradeoffAn object store by default, the database for small blobs

We gain

The cheapest durable GB, a small database that fails over in seconds, and scaling for bytes you never operate.

We give up

Every write spans two systems, with an ordering rule and a reconciler. And every read pays an extra hop, unless a CDN or cache absorbs it.

How uploads reach the store

Upload path	Wins when	Costs
Through the API server	Bytes must be inspected or transformed in line, small files only	App servers carry every byte and every slow client
Presigned single PUT	Files up to tens of MB on decent networks: avatars, documents	Restart from zero on failure, no size cap on the URL
Presigned multipart	Large files, mobile clients, anything over about 100 MB	Part tracking, a complete call, an abort lifecycle rule
Resumable session (GCS, tus)	Very unreliable networks, clients that restart, browser uploads	Server-side session state, a protocol to implement on both ends

TradeoffPresigned multipart for media, single PUT for small files

We gain

A dropped connection costs at most the parts in flight, parts upload in parallel, and small files stay a single request.

We give up

Two client code paths, a billed PUT per part, resume logic, and invisible billed parts if nothing aborts abandoned uploads.

Storage classes and lifecycle

Reads skew heavily by age: a photo gets most of its views in its first days. Storage classes trade storage price for retrieval price and latency, and lifecycle rules move objects between them by age automatically. Approximate S3 list prices in us-east-1 (they change, and fall slightly at volume):

Class	Storage, per TB-month	Retrieval	First byte	Minimum stay	Fits
Standard	$23	Free	Tens of ms	None	New, hot and unknown data
Standard-IA	$12.50	$10 per TB	Tens of ms	30 days	Read about once a month: older media
One Zone-IA	$10	$10 per TB	Tens of ms	30 days	Re-creatable data (thumbnails you can rerender)
Glacier Instant Retrieval	$4	$30 per TB	Tens of ms	90 days	Read a few times a year, but instantly
Glacier Flexible Retrieval	$3.60	Varies	Minutes to 12 hours	90 days	Backups, compliance copies
Glacier Deep Archive	About $1	Varies	Within 12 to 48 hours	180 days	Legal holds, data you hope never to read

The shape matters more than the exact prices. Down the ladder, storage falls from $23 to about $1 per TB-month. Each step down adds a retrieval fee and a longer minimum duration. At the bottom, it adds hours of latency.

Deleting early still bills the minimum. The infrequent access and Glacier Instant classes bill objects under 128 KB as 128 KB. And every transition is a billed request per object. So tiering a billion small thumbnails can cost more than it saves.

A worked example. 1 PB of photos, where 20% is younger than 30 days, 30% is 30 to 90 days old and 50% older:

A lifecycle policy for user media

Scroll or pinch to zoom, drag to pan

• Color key
• Database
• Failure

Layout	Monthly storage bill
All 1 PB in Standard	About $23,000
200 TB Standard, 300 TB IA, 500 TB Glacier IR	About $4,600 + $3,750 + $2,000 = $10,350

That halves the bill, as long as the old photos really are rarely read. Reading back 100 TB a month from Glacier Instant Retrieval costs $3,000 in retrieval fees. When nobody knows the access pattern, use Intelligent-Tiering. It moves each object between tiers by its own access history, for a small monitoring fee per object. (Objects under 128 KB are not tiered.)

TradeoffAge-based lifecycle rules for media

We gain

About half the storage bill for data whose reads fall off with age, with no application change.

We give up

Retrieval fees and minimum durations punish a wrong guess about access. Archive tiers take hours to read. And small objects can cost more to move than they save.

Which object store

Rent one, unless the bill or the law says otherwise. At hundreds of PB, the provider’s margin outgrows the cost of a storage team. Dropbox moved most of its data off S3 to its own store in 2016, as the file sync problem discusses.

Go deeperManaged, zero-egress or self-hosted1 min

Option	Wins when	Watch out for
S3, GCS, Azure Blob	The default: you run on that cloud, need the ecosystem (events, lifecycle, IAM)	Egress fees, per-request fees, lock-in to one API’s extras
Zero-egress stores (Cloudflare R2 and others)	Read-heavy public content served to the internet	Fewer storage classes and integrations; check the request pricing
Self-hosted (Ceph, MinIO and similar)	On-premises, data sovereignty, or exabyte scale where the margin is the bill	You now run durability, repair, scrubbing and capacity planning yourself

TradeoffA managed store until the bill says otherwise

We gain

Durability, scaling, lifecycle and events on day one, with no storage engineers.

We give up

Egress and request fees grow with success, and moving petabytes out later takes months.

How it fails

Orphans, dangling rows and partial uploads

The two-store seam from step 4 fails four ways:

• A dangling row: metadata committed first, or an object deleted before its row.
• An orphan object.
• A stuck pending row.
• An abandoned multipart upload.

Only the first hurts users. So prevent it with the write order, and delete in reverse: hide the row first, and delete the object later. The reconciler, a sweeper and the abort rule handle the other three.

Event-driven processing adds one more. S3 event notifications are delivered at least once. They usually arrive within seconds, but sometimes later, and not always in order. So a thumbnail worker must be idempotent. (The Instagram pipeline claims each media row with a state check.) And the reconciler is the backstop for an event that never arrives.

Hot objects and 503 Slow Down

S3 splits a busy prefix automatically, but the split takes minutes, and a sudden burst past about 3,500 writes a second gets 503 Slow Down meanwhile. Writes clustered under one prefix (logs/2026-10-28/... from every server) hit this first, so put a high-cardinality part early in the key. A single object cannot be split at all: a viral video read 50,000 times a second is throttled and billed for egress. A CDN with an origin shield turns those reads into a handful, as in the pastebin problem’s viral paste.

Accidental public buckets

Misconfigured public buckets are a recurring cause of large data leaks. A typical case: a read-for-everyone policy, set once for a static site, on a bucket that later receives user documents. Three defenses:

• Turn on Block Public Access at the account level. It’s the default for new S3 buckets since April 2023, but not for older ones.
• Keep one purpose per bucket, so public assets never share a bucket with user data.
• Run configuration scanning that pages on any public grant.

Unguessable keys are not access control. A random key is found the first time its URL is shared or logged.

Deletes, overwrites and versioning

A buggy cleanup job, a script run against the wrong bucket, or ransomware with stolen credentials erases data with full eleven-nines reliability. The defenses:

• Versioning: a DELETE adds a delete marker and the old version can be restored. Old versions are billed, so expire them after 30 to 90 days with a lifecycle rule.
• Object Lock (write once, read many) blocks deletion for a retention period, even by the account root in compliance mode. Use it for backups and audit logs.
• A copy in another account, with separate credentials, survives the first one’s compromise.
• Circuit breakers in anything that deletes in bulk, like the reconciler in step 4.

Go deeperEvery failure mode at a glance1 min

Failure	What breaks	How you see it	Blast radius	Fix
Metadata committed first	Rows point at missing objects	Broken-image reports, inventory join	Every upload that failed midway	Bytes first or a pending row; reconciler
No abort lifecycle rule	Parts of abandoned uploads billed forever	Storage Lens or bill vs LIST total	Cost only, grows quietly	AbortIncompleteMultipartUpload after 7 days
Event lost or duplicated	Thumbnails missing, or processed twice	Rows stuck in uploaded, duplicate work counts	Individual uploads	Idempotent workers, sweeper for stuck rows
Burst on one prefix	503 Slow Down on writes	503 rate per prefix	Writes under that prefix	Spread keys, retry with backoff and jitter
Viral object	Throttled reads and a large egress bill	Per-key request rate, egress spike	One object, the bill	CDN with origin shield
Public bucket	User data readable by anyone	Config scanning, access logs from unknown IPs	Everything in the bucket	Block Public Access, one purpose per bucket
Mistaken or malicious delete	Data gone, correctly and durably	Delete rate alarms	Up to the whole bucket	Versioning, Object Lock, copy in another account
Billions of tiny objects	Request fees dwarf storage, lifecycle costs more than it saves	PUT and transition request line items	The bill	Pack small blobs into segments, keep them in Standard

Common mistake

Eleven nines means my data is safe

Eleven nines describes the chance that the store loses an object through hardware failure. It says nothing about your application deleting the wrong objects. Or a reconciler joining against an empty table, a lifecycle rule with the wrong prefix, or an attacker with your credentials. Those are where data is actually lost. Versioning, Object Lock and an independent copy protect against them. More nines do not.

At scale

Number	Value
Time to first byte, same region	About 10 to 100 ms; single-digit ms on low-latency classes (S3 Express One Zone)
Throughput per connection	About 50 to 100 MB/s; parallel ranges or parts for more
Request rate per prefix partition	About 3,500 writes and 5,500 reads a second
Largest single PUT; multipart parts	5 GB; 5 MB to 5 GB each, up to 10,000
Requests	About $5 per million PUTs, $0.40 per million GETs
Internet egress from the bucket	About $90 per TB at low volume, less with a CDN and at scale

At 10 times the traffic. The store scales without you, so the pressure moves to its edges. Uploads move fully to presigned multipart. Reads move behind a CDN, because egress becomes the largest line on the bill. Write-heavy keys get a high-cardinality prefix. The reconciler moves to daily inventory reports. And lifecycle rules start to matter, as storage grows by hundreds of TB a year.

At 100 times. Cost becomes the design. Small objects get packed into segments, tiering follows measured access, renditions of cold media are re-rendered on demand, and other regions get only the originals. At hundreds of PB a self-built, erasure-coded store (f4 at Facebook, Magic Pocket at Dropbox) beats the provider’s price, and the team now owns durability.

What to monitor. Bytes stored and object count per bucket and class, against the metadata table’s totals (a growing gap means orphans or dangling rows). Bytes held in incomplete multipart uploads. 4xx and 5xx rates per operation, with 503s per prefix. p99 time to first byte. CDN hit ratio and origin egress. Rows stuck in uploading past an hour. Delete rates, with an alarm on any spike. And any bucket policy change that grants public access.

In the interview

Interviewer asksUsers upload videos up to 2 GB from phones on bad networks. Walk me through the upload path.

The client asks the API to start an upload. The API checks quota, and inserts a row in state uploading with a fresh key. It calls CreateMultipartUpload, and returns the upload id with presigned URLs for, say, 8 MB parts. The phone PUTs parts directly to the object store, four at a time. If the connection drops, it asks which parts arrived (ListParts), and sends only the rest. When all parts are in, the API calls CompleteMultipartUpload, which makes the object appear atomically. It checks size and checksum, and flips the row to uploaded. That emits an event for transcoding. A lifecycle rule aborts abandoned uploads after 7 days. A sweeper fails rows stuck in uploading.

Interviewer asksThere is no transaction across your database and S3. How do you keep them consistent?

I order the writes so a crash can only leave a harmless inconsistency. First a hidden pending row, then the bytes, then the flip to ready. A crash leaves an orphan object, never a row pointing at nothing. Deletes run in reverse: hide the row, then delete the object later. A daily reconciler joins the bucket inventory with the table. It deletes orphans older than a day, and settles stuck rows. It has a circuit breaker, so a bad read of the table cannot delete the bucket.

Interviewer asksStorage is growing by 700 TB a month. How do you bring the bill down?

First, split the bill into storage, requests and egress. For storage, a lifecycle policy roughly halves the cost of media whose reads fall off with age: Standard to Infrequent Access at 30 days, and Glacier Instant Retrieval at 90. Check retrieval rates first, because retrieval fees punish a wrong guess. Then stop storing what can be re-created. Expire old versions, abort incomplete uploads, and dedup by content hash. If requests dominate, pack small objects. If egress does, raise the CDN hit ratio.

Common mistake

Uploading through the API server

“The client sends the file to our API, which writes it to S3” puts every byte, and every slow mobile connection, on the app fleet. The fleet must now be sized for bandwidth and connection time, instead of requests. Hand out presigned URLs, and let the client upload straight to the store. Then validate asynchronously.

Mid-level covers

Puts files in S3 and metadata in a database, uses presigned URLs, mentions a CDN for reads and multipart for large files.

Senior adds

Orders the writes so a crash leaves only orphans, and names the reconciler. Signs content type and checksum, and knows a presigned PUT cannot set a size range. Sizes parts against the 10,000 part limit, and adds the abort rule. Compares replication and erasure coding with overhead numbers. Prices tiers with their retrieval fees and minimums. Knows S3 has been strongly consistent since 2020. Guards against deletes and public buckets.

Seen in

• Pastebin: paste bodies in an object store under random keys. Body first, metadata second, with a reconciler that deletes orphans from the daily inventory.
• Instagram: presigned multipart uploads from phones, an object-created event that drives rendering, and erasure-coded warm storage for media older than 90 days.
• File sync: content-addressed blocks of about 1.25 MB, blocks first and metadata last, mark-and-sweep garbage collection and dedup that does not leak.
• Video streaming: resumable multipart uploads in 8 MB parts, and immutable segments served through a CDN and an origin shield.
• News feed: media uploaded straight to blob storage with a presigned URL before the post is created.
• Chat: attachments through presigned uploads, and older message history moved to object storage.
• Collaborative editor: document snapshots stored as immutable objects keyed by document and revision.
• Message queue: tiered storage, with closed log segments uploaded to object storage and brokers keeping only recent hours.
• Metrics and monitoring: two-hour TSDB blocks on object storage, compacted offline, with caches in front of slow reads.
• Web crawler: pages packed into 1 GB WARC files to avoid per-object request costs, and frontier checkpoints written to S3.

Recap

The job. Keep billions of immutable blobs cheaply and durably, by key. The database keeps the small metadata row. The object store keeps the bytes.

The model. Flat keys, immutable objects, no rename, slow paged LIST, no multi-object transactions, fees per request. Strongly consistent per object in S3 since December 2020.

Durability. Eleven nines from erasure coding across zones: RS(10,4) survives 4 losses at 1.4 times the data, against 3 times for three copies. Fast repair matters as much as the code.

The seam. Bytes first, metadata last, or a hidden pending row flipped last. A reconciler with a circuit breaker removes orphans and settles stuck rows.

Uploads. Presigned URLs keep bytes off app servers. Sign the method, key, content type and checksum, with a short expiry. Use multipart with resume for big files, plus an abort lifecycle rule.

Reads and cost. Immutable keys behind a CDN, bucket private. Lifecycle tiers halve storage for aging data but charge for retrieval. Pack tiny objects.

Biggest risks. Rows pointing at missing bytes, invisible abandoned uploads, a hot object or prefix, a public bucket, and a mistaken delete executed perfectly.

Further reading

• On this site: CDNs for the read path, databases for the metadata side, and replication for copies and quorums.
• Amazon S3 performance guidelines: request rates per prefix, byte-range fetches and parallel uploads, from AWS.
• f4: Facebook’s Warm BLOB Storage System, OSDI 2014: Reed-Solomon (10,4) for warm photos and the effective replication factor.
• Finding a Needle in Haystack: Facebook’s Photo Storage, OSDI 2010: packing billions of small photos into large files.