Idempotency and exactly-once effects

An operation is idempotent when doing it twice leaves the system in the same state as doing it once. Every distributed system retries, because a timeout cannot tell “it failed” from “it worked and the answer got lost”. So every step that a retry can reach must be idempotent, or the system charges twice, emails twice and counts twice. Exactly-once effects are what you get when at-least-once delivery meets idempotent steps at every hop.

Single hops have their own pages:

• API design, for the Idempotency-Key header and key table.
• Message queues, for the idempotent consumer.
• Distributed transactions, for the outbox and inbox.

This page covers the whole path.

The problem it solves

A buyer taps Pay once on a $40 order. The request crosses six hops before the money moves and the receipt arrives:

1. The app.
2. A load balancer.
3. The payments service, with its outbox.
4. A queue and a worker.
5. The payment provider.
6. The provider’s webhook back to us, which sends the email.

Each hop can repeat what it received:

One tap, and every place it can be repeated

Scroll or pinch to zoom, drag to pan

• Color key
• Service
• Database

Source of the repeat	Why it happens	What it repeats
Double tap or client retry	Impatient thumb; mobile timeout; offline queue replayed hours later	The whole request
Load balancer or mesh retry	Upstream slow or reset; a retry policy that covers POST	A request, sometimes in parallel
Outbox relay or producer retry	Crash after publishing, before marking sent; lost broker ack	A message, with the same id
Queue redelivery	Consumer crash, visibility timeout, rebalance	A message, possibly while it still runs
Consumer crash after effect	The effect happened, the record of it did not commit	The side effect
Call to a third party	Reply lost; our retry loop	A charge, a shipment, an SMS
Webhook redelivery	Our handler answered slowly or with 5xx; Stripe retries for up to 3 days	An event, with the same event id
Scheduler double run	Failover, two replicas, clock jumps; Kubernetes CronJob docs warn of exactly this	A whole batch
Deliberate replay	Reprocessing a log after a bug fix, a dead-letter redrive	Everything in the replayed range

The rates look small. Assume each of five hops repeats 0.1% of what it handles. That’s ordinary for a mobile client and a busy queue. Without guards, about 0.5% of payments go through twice. At a million orders a day, that is about 5,000 double charges a day. Each one is a support ticket, a refund, and possibly a chargeback fee.

What the system must deliver is an observable property. One tap produces one payment row, one charge and one receipt, whatever was repeated on the way.

Idempotent is not the same as safe. A safe call (GET) changes nothing. An idempotent call (PUT, DELETE) changes state the same way, however many times it runs. And idempotency is about state, not the response. A second DELETE may return 404 where the first returned 204. RFC 9110 still calls it idempotent. The caller must treat “already done” as success.

How it works

Step 1: Accept that every hop repeats

A sender that hears nothing back has two choices. It can send again (at least once), or give up (at most once). From its side, a lost request and a lost reply look the same. So no protocol removes that choice. Exactly-once delivery between two machines does not exist. (Message queues shows why.) For anything that matters, choose at least once, and make the receiver absorb repeats:

effectively once = at-least-once delivery + an idempotent effect at every hop

Three tools absorb repeats. In order of preference: make the operation idempotent by nature, make the write conditional on state, or remember what you have done under a key. A fourth tool, fencing, stops a worker that no longer has the right to act.

Step 2: Make the operation idempotent by nature

The cheapest repeat is one that changes nothing, because of how the write is phrased. A delta is not idempotent. An absolute value is.

Not idempotent	Idempotent version	Why it works
balance = balance - 40	Insert ledger line with id pay_91:debit, unique	The line exists once; balance is a sum
INCR clicks:ad7	clicks:ad7:14:31 = 413 (the full total)	Writing 413 twice leaves 413
ZINCRBY board 50 player9	ZADD board GT 1250 player9	Absolute total, and never moves backwards
POST /orders (server picks id)	PUT /orders/{client-uuid}	The second insert hits the same key
INSERT INTO likes ...	INSERT ... ON CONFLICT (user, post) DO NOTHING	A like is a member of a set
“Append chunk to file”	PUT /blocks/{sha256}	Content-addressed: same bytes, same name

Commutative writes (set membership, max) also survive reordering, which redelivery and replays cause. Some absolute values only grow, like the totals in the ad click aggregator. A sink can keep the larger of two writes. Then a replayed older total cannot move the number backwards.

“Charge the card” and “send the email” are deltas by nature. No rephrasing makes them states, so they need the next tools.

Step 3: Make the write conditional on state

Most business entities move through a state machine: an order is placed, packed, shipped, delivered. A transition can carry its own guard:

UPDATE orders SET status = 'PACKED', version = version + 1
WHERE id = 'o1' AND status = 'PLACED';

The first delivery matches one row. Every repeat matches zero, because the row is no longer PLACED. The database applies the statement atomically per row, so two workers that both read PLACED cannot both win. There is no dedupe table and no expiry: the row’s own state remembers that the step happened for as long as the order exists.

The detail most code gets wrong is what to do with zero rows. Zero rows means one of two very different things:

• The step already happened: a redelivery, a twin worker that won, or a late copy that arrives after the order moved on. The caller should hear success, ack the message and stop retrying. Raising an error here sends a harmless repeat to the dead-letter queue, or worse, makes a client retry forever.
• The order went somewhere else, such as CANCELLED, or this message arrived before the step it follows. The caller must decide (retry later, refund, alert). Reporting success would hide a bug.

Tell them apart by reading where the row is now:

JMain.javaPyorders.pyC#Program.csGoorders.goTSorders.ts

JavaPythonC#GoTypeScript

JavaPythonC#GoTypeScript

import java.util.HashMap;
import java.util.List;
import java.util.Map;

enum Outcome {
    APPLIED, // this call made the change
    ALREADY_DONE, // a repeat or a late event: the change happened before
    CONFLICT // the row went somewhere this step does not lead
}

final class Row {
    String status;
    int version; // bumped only by a real change, so readers can tell repeats did nothing

    Row(String status, int version) {
        this.status = status;
        this.version = version;
    }

    @Override
    public String toString() {
        return "Row(status=" + status + ", version=" + version + ")";
    }
}

// Stands in for a table. compareAndSet is one statement,
// UPDATE orders SET status = :to, version = version + 1 WHERE id = :id AND status = :from,
// and the database applies it atomically per row: two workers that both read PLACED cannot
// both win, whichever machine they run on.
final class Orders {
    final Map<String, Row> rows = new HashMap<>();

    void insert(String orderId, String status) {
        rows.put(orderId, new Row(status, 1));
    }

    boolean compareAndSet(String orderId, String expected, String next) {
        Row row = rows.get(orderId);
        if (!row.status.equals(expected)) return false; // zero rows matched
        row.status = next;
        row.version++;
        return true;
    }

    // UPDATE orders SET status = :to WHERE id = :id. Idempotent for one message repeated,
    // wrong as soon as messages arrive out of order.
    void blindSet(String orderId, String next) {
        rows.get(orderId).status = next;
    }
}

final class Shipments {
    // The forward path of a shipment. A repeat of a step finds the row at or past that step's target.
    static final List<String> PATH = List.of("PLACED", "PACKED", "SHIPPED", "DELIVERED");

    static Outcome advance(Orders db, String orderId, String from, String to) {
        // The WHERE clause is the whole idempotency mechanism. No dedupe table and no TTL: the
        // row's own state remembers that the step happened, for as long as the order exists.
        if (db.compareAndSet(orderId, from, to)) return Outcome.APPLIED;
        String current = db.rows.get(orderId).status;
        // Zero rows matched. Either this step already happened (a duplicate, or a twin worker
        // beat us), or the order went somewhere else. Tell them apart by where the row is now.
        if (PATH.contains(current) && PATH.indexOf(current) >= PATH.indexOf(to)) {
            // Report success. An error here makes the caller retry, or dead-letter a message
            // that was only a harmless repeat.
            return Outcome.ALREADY_DONE;
        }
        // CANCELLED, or a step that arrived before the one it follows: the caller decides (retry later,
        // refund, alert), never a silent success.
        return Outcome.CONFLICT;
    }
}

class Main {
    public static void main(String[] args) {
        Orders db = new Orders();
        db.insert("o1", "PLACED");
        System.out.println(Shipments.advance(db, "o1", "PLACED", "PACKED")); // APPLIED
        // The queue redelivers the warehouse message after a consumer crash.
        System.out.println(Shipments.advance(db, "o1", "PLACED", "PACKED")); // ALREADY_DONE
        System.out.println(Shipments.advance(db, "o1", "PACKED", "SHIPPED")); // APPLIED (carrier webhook)
        System.out.println(Shipments.advance(db, "o1", "PACKED", "SHIPPED")); // ALREADY_DONE (sent again)
        System.out.println(Shipments.advance(db, "o1", "SHIPPED", "DELIVERED")); // APPLIED
        // A delayed copy of the first message arrives last. The condition keeps the order DELIVERED.
        System.out.println(Shipments.advance(db, "o1", "PLACED", "PACKED")); // ALREADY_DONE
        System.out.println(db.rows.get("o1")); // Row(status=DELIVERED, version=4)

        // Two workers hold the same message and both read PLACED. The database runs their
        // UPDATEs one after the other, so exactly one matches a row.
        db.insert("o2", "PLACED");
        System.out.println(Shipments.advance(db, "o2", "PLACED", "PACKED") + " "
                + Shipments.advance(db, "o2", "PLACED", "PACKED")); // APPLIED ALREADY_DONE

        db.insert("o3", "CANCELLED");
        System.out.println(Shipments.advance(db, "o3", "PLACED", "PACKED")); // CONFLICT

        // The same late message, written blind, moves a delivered order back to PACKED.
        db.blindSet("o1", "PACKED");
        System.out.println(db.rows.get("o1").status); // PACKED: "idempotent" and still wrong
    }
}

The last lines of the run show a subtle bug. A plain SET status = 'PACKED' is idempotent for one repeated message. But it’s still wrong, because a late copy of an old message moves a delivered order backwards. A condition on the current state survives both repeats and reordering.

There are two variants of the same idea. A version column (WHERE version = 7) suits read-modify-write edits that are not a fixed state machine. A per-entity sequence from the producer is applied only if seq > last_applied_seq. The sequence costs one number per entity, instead of one row per message. That matters at scale (see the numbers below).

TradeoffConditional writes as the first choice for state changes

We gain

Idempotent and reorder-safe with no extra table, no TTL and no window to get wrong. The database enforces it, so two workers racing on two machines are handled by the same statement.

We give up

Only works when the effect is a change to a row you own. A response the caller needs again (a decline code) is not stored, and an external effect, like a charge, sits outside the condition.

Step 4: Remember what you did, under a key, for a window

When neither fits, the receiver records a key per operation, and checks it before acting. That’s the key table on API design, and the inbox on message queues. The same four rules hold at every hop:

1. Insert the key first, under a unique index.
2. Commit it with the effect.
3. Store the outcome, for replay.
4. Mark it in flight, with a lease.

The open question is how long to remember. A copy that arrives after its key expired is not recognized, and runs again. So:

dedupe window > the longest possible gap between the first copy and the last copy

The gap is set by the slowest path a repeat can take, not by the typical retry:

Where the late copy comes from	How late it can be
Client retry with backoff	Seconds to minutes
Mobile offline queue	Hours, sometimes days
Webhook retries (Stripe, live mode)	Up to 3 days, with exponential backoff
Kafka replay from the earliest offset	The topic’s retention: 7 days by default
SQS dead-letter redrive	Up to the DLQ’s retention, at most 14 days
A deliberate backfill after a bug fix	As far back as you replay: weeks
SQS FIFO’s own deduplication	Only catches repeats within 5 minutes of the first

The last row is the counter-example. A broker’s built-in dedupe is a short window, aimed at producer retries. It does nothing for a replay an hour later.

Three ways out of the window problem, in order of preference:

• Use a key that lives as long as the entity. A unique constraint on (order_id, step) in the payments table never expires, because the payment row does not. That’s why business keys beat random request keys, wherever one exists.
• Use a per-entity version (step 3): one number per entity, kept forever, cheap.
• Size the TTL to the slowest path. And send the slowest paths (replays, redrives) through a check that does not depend on the window, such as reconciliation against the source of truth.

Go deeperProbabilistic pre-checks for very large dedupe sets1 min

A consumer seeing tens of thousands of events a second can put a Bloom filter in front of its dedupe store. “Definitely not seen”, the common case, skips the store lookup. “Maybe seen” goes to the store for the exact answer.

Never let the filter decide alone. A false positive would drop a real event as a duplicate, which is a silent loss. The filter saves reads, not correctness. Stream processors such as Flink keep dedupe keys in partitioned state with a TTL instead. That’s local and fast, but has the same window rule.

Step 5: Fence writes from workers that lost the right to act

Some duplicates come from a worker that does not know it was replaced. It held a lease on a job. It paused for 40 seconds in garbage collection, and the lease expired. Another worker took the job. Then the first one wakes up, and finishes its write. Both writes are “the same operation”. But a key alone does not help if the second writer’s effect is different, because it read newer data.

A fencing token fixes it. Every grant of the lease comes with a number that only grows. The resource refuses writes carrying a number lower than the highest it has seen:

UPDATE runs SET result = :r, fence = :token
WHERE id = :run AND fence <= :token;

The consensus page covers where the tokens come from, and the job scheduler draws the paused worker. Kafka uses the same idea for producers: each transactional.id has an epoch, and a restarted producer bumps it, so a zombie instance’s writes are rejected.

Fencing and idempotency keys solve different halves. The key makes the same operation harmless to repeat. The fence stops an out-of-date actor from writing at all. A job that charges a card needs both: the run id as the provider’s key, and the token on writes to our own store.

Step 6: Choose what the key identifies

The key decides what counts as “the same operation”. It must identify the user’s intent, not the HTTP attempt and not the payload.

Key source	Example	Right when	Breaks when
Client-generated, per intent	UUID made when the checkout screen opens	The client starts the operation	The client makes a new one per attempt, or loses it on app restart
Derived from a parent key	sha256(order_id + "/charge/1")	Every hop after the first	The derivation leaves out something that makes two real operations differ
Business identity	(merchant_id, merchant_order_ref), clOrdId	A natural unique id exists	It is not truly unique (two legitimate rows share it)
Event id from the source	Stripe evt_..., a Kafka record’s event id	Consuming someone else’s events	The producer mints a new id when it republishes
Scheduled slot	nightly-invoices@2026-10-28T02:00Z	Cron and periodic jobs	Built from the clock at start, so two replicas get two keys
Content hash	sha256(user, amount, merchant)	Immutable content: file blocks, uploaded objects	Two genuine identical requests: the second is swallowed

Content hashes deserve their warning. They are perfect for content-addressed storage, where two identical blocks really are one block, as in file sync. For actions they are a bug: a buyer who orders the same $4.50 coffee twice in ten minutes made two purchases with identical content.

Derivation is how one key from the client becomes a key at every later hop without storing each one first. Hash the parent key with the step name. Any retry, on any machine, after any crash, derives the same child key:

JKeys.javaPyexample.pyC#Keys.csGomain.goTSexample.ts

JavaPythonC#GoTypeScript

JavaPythonC#GoTypeScript

import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.util.HexFormat;

final class Keys {
    static String sha256(String s) {
        try {
            byte[] d = MessageDigest.getInstance("SHA-256").digest(s.getBytes(StandardCharsets.UTF_8));
            return HexFormat.of().formatHex(d).substring(0, 24);
        } catch (NoSuchAlgorithmException e) {
            throw new IllegalStateException(e); // every JVM ships SHA-256
        }
    }

    // The same parent and step give the same key on every retry, on any machine, after any
    // crash, without storing it first. Different steps of one intent never share a key.
    static String childKey(String parent, String step) {
        return sha256(parent + "/" + step);
    }

    // The key sent to the payment provider. A retry of attempt 1 reuses attempt 1's key. A new
    // attempt number is a deliberate new operation: the buyer picked another card after a decline.
    static String chargeKey(String paymentId, int attempt) {
        return childKey(paymentId, "charge/" + attempt);
    }

    // The slot the run belongs to, never the clock when it started. Two schedulers that both fire
    // the 02:00 run started at 02:00:00.4 and 02:00:03, and must still produce one key.
    static String cronRunKey(String job, long scheduledFor) {
        return job + "@" + scheduledFor;
    }

    // The tempting shortcut: hash what the request says. Two genuine identical purchases have the
    // same content, so the second one would be swallowed as a "duplicate".
    static String contentKey(String userId, int amountCents, String merchant) {
        return sha256(userId + "|" + amountCents + "|" + merchant);
    }
}

class Main {
    public static void main(String[] args) {
        // The root key is made once per user intent, when the checkout screen opens, and kept on
        // the device until the intent finishes. Every key below is derived from it.
        String root = "chk_7f3a9c";
        String order = Keys.childKey(root, "order");
        System.out.println(order.equals(Keys.childKey(root, "order"))); // true: same order key on retry

        String payment = Keys.childKey(order, "payment");
        System.out.println(Keys.chargeKey(payment, 1).equals(Keys.chargeKey(payment, 1))); // true
        System.out.println(Keys.chargeKey(payment, 1).equals(Keys.chargeKey(payment, 2))); // false: new card
        System.out.println(Keys.childKey(order, "email/receipt").equals(Keys.childKey(order, "email/shipped"))); // false

        // Both scheduler replicas act on the run due at 2026-10-28T02:00:00Z.
        long slot = 1_793_152_800L;
        System.out.println(Keys.cronRunKey("nightly-invoices", slot).equals(Keys.cronRunKey("nightly-invoices", slot))); // true

        // Two coffees, same price, same shop, ten minutes apart: one key, one lost sale.
        System.out.println(Keys.contentKey("u_42", 450, "m_cafe").equals(Keys.contentKey("u_42", 450, "m_cafe"))); // true
    }
}

The attempt number in the charge key is deliberate. A retry of a call reuses its key. But say the card was declined, and the buyer chose another card. That second attempt is a new operation, and gets a new key. Otherwise the provider would replay the decline forever. The payment system stores an attempt id per call for exactly this reason.

TradeoffClient key at the edge, derived keys inside

We gain

One decision, made once where the intent starts, protects every hop. Keys are recomputed instead of looked up, so a crash between two hops cannot lose them.

We give up

The client must persist the key until the intent finishes, including across app restarts, and every service must agree on the derivation. A new attempt must be an explicit decision, never an accident.

Step 7: Carry the key across every hop, including other companies

Your transaction cannot cover a system you do not own. Say the effect happens at a payment provider, a carrier or a cloud API. Then the only way to make the repeat harmless is for them to recognize it. So the key has to travel:

One intent, one key per hop

Scroll or pinch to zoom, drag to pan

• Color key
• Service
• Background worker

Most payment and cloud APIs accept a key: Stripe’s Idempotency-Key header, PayPal’s PayPal-Request-Id, the ClientToken on AWS EC2’s RunInstances. Check two things for each:

• Retention. Stripe may prune keys once they are 24 hours old. A retry after that is a new request. So past the provider’s window, never retry blind. Look up the outcome by your own reference (the payment id you sent as metadata), and act on what you find. The payment system’s resolver does this.
• Scope. A key is usually scoped to your account, and sometimes to one endpoint. Two different operations must never share a key, or the second gets the first one’s result and silently does nothing.

Some downstream APIs have no idempotency support. Then persist the intent before calling (“charge requested for P1”). After any unknown outcome, query the downstream before deciding. Check, then act.

The whole path, hop by hop, each with its classic failure and its guard. Start with the lost response and no guards, then switch guards on one at a time:

All guards offAll guards onBreak every hopNo failures

1. 1App
2. 2Load balancer
3. 3Service and outbox
4. 4Queue and worker
5. 5Payment provider
6. 6Webhook and email

1App

BreakResponse lost, app retriesGuardSame key on every retry

Quiet

2Load balancer

BreakUpstream slow, balancer retriesGuardNever retry a POST

Quiet

3Service and outbox

BreakRelay publishes the job twiceGuardKey table, unique insert first

Quiet

4Queue and worker

BreakWorker crashes after chargingGuardInbox of message ids

Quiet

5Payment provider

BreakReply lost, worker retriesGuardIdempotency key sent to it

Quiet

6Webhook and email

BreakWebhook delivered twiceGuardDedupe by event id

Quiet

Payment rows

1

Card charges

1

Receipt emails

1

Step by step

Things to try:

• Guard only the service. The key table misses a retry that carries a fresh key. Turn on “same key on every retry” and the duplicate disappears.
• Break the balancer and turn on “Never retry a POST”. The duplicate moves to the app, which gets a 504 and retries. Refusing to retry at one layer hands the decision to another.
• Break the worker, inbox on, provider key off. The crash rolled back the inbox row, but the charge had already happened. Only the key sent downstream stops the second charge.
• Break every hop, guard only the webhook. Six charges, six receipts: six different events, so dedupe by event id lets each through. Guards belong where the repeat starts.

Step 8: Bound the effects you cannot make idempotent

Some effects have no key, no state to condition on, and no undo. Think of an email, an SMS, a push notification, a physical action (a parcel shipped, cash dispensed), or a call to an API that ignores retries. There’s a window between “the effect happened” and “we recorded that it happened”. It cannot be closed. It can only be made small, rare, and harmless.

The first decision is the direction of failure:

Order	After a crash between the two steps	Right for
Record “sending”, then send (claim first)	The message may be lost (at most once)	Marketing, digests, nice-to-have nudges
Send, then record “sent”	The message may repeat (at least once)	Login codes, security alerts, receipts

Then shrink and soften the duplicate:

• Claim with a lease, not a lock. A claim on (notification_id, channel) with a 2 minute expiry stops a redelivery while the first sender is alive. And it allows a retry if the sender died (notification system).
• Use the provider’s collapse feature. APNs apns-collapse-id and FCM collapse_key make a second push replace the first on the device.
• Make the content idempotent. A repeated OTP SMS carries the same code while it is valid. “Your order 42 shipped” survives a repeat. “You have 1 new message” does not.
• Cap per recipient. “At most 3 transactional emails per order per hour” bounds a retry storm.
• Put irreversible steps last. In a saga, the pivot and the physical action come after everything that can still fail.

Step 9: Decode “exactly once” claims

Products advertise exactly-once in specific, narrow senses. Each one is real, and each one stops at a boundary:

Claim	What it really guarantees	What it does not cover
Kafka idempotent producer (default since 3.0)	Retries from one producer session do not duplicate or reorder records in a partition (producer id plus sequence)	A restarted producer without a transactional id; a relay that republishes; any consumer
Kafka transactions (exactly-once semantics)	Read from Kafka, write to Kafka and commit offsets atomically; readers use read_committed	Any effect outside Kafka: a database row, an email, a card charge
Flink end-to-end exactly-once	State restored from checkpoints, plus a sink that commits with the checkpoint (two-phase) or is idempotent	Sinks that do neither; duplicates already in the input
SQS FIFO	Drops a repeated send with the same deduplication id within 5 minutes	A consumer that acts and then dies before deleting the message
Google Pub/Sub exactly-once delivery	Pull subscriptions, within one region: no redelivery after an ack succeeds or while the ack deadline holds	Push subscriptions; an effect performed before an ack that then fails

Read the right column, and the pattern is the same. The guarantee covers the system’s own records. The moment work leaves it (a row in your database, a provider call), you are back to at-least-once. And the step has to be idempotent. The message queue problem builds Kafka’s mechanism. The point here is where it ends.

So state the guarantee as an observable property instead. For example: “each payment intent produces at most one successful charge, and every charge is reflected in exactly one ledger entry”. That can be tested and reconciled.

Options and tradeoffs

Which tool for which hop

Tool	How a repeat is absorbed	State kept	Survives reordering	Wins when
Natural idempotence	The write is a state, not a delta	None	If commutative too	Counters as totals, sets, upserts, content blobs
Conditional write	WHERE status = :from or version = :v	The entity’s own state	Yes	State machines on rows you own
Per-entity sequence	apply if seq > last_seq	One number per entity	Yes	Event streams with an ordered producer per entity
Unique business key	Second insert fails the constraint	A row that lives with the entity	Yes	One row per operation, natural id available
Dedupe table (key, outcome)	Key lookup, stored outcome replayed	A row per operation, with a TTL	Yes, within the window	Arbitrary operations, responses to replay
Fencing token	Writes with an older token refused	Highest token per resource	Yes	Leased work where a paused actor may wake up
Downstream key	The other system recognizes the repeat	Theirs, with their retention	Depends on them	Effects in someone else’s system

TradeoffPrefer state over memory

We gain

Natural idempotence and conditional writes need no extra storage, never expire and handle reordering. A dedupe table is kept for the cases they cannot express.

We give up

Writes must be designed as states and transitions from the start. An existing API built on deltas (+1, -40) needs reshaping before it can use them.

Go deeperWhere the dedupe record lives, and what a repeat gets back2 min

Option	Atomic with the effect	Cost	Wins when
Same database, same transaction	Yes	A row and an index entry per operation	The effect is a write to that database
Redis SET key NX PX ttl	No	About 60 to 100 bytes per key, in RAM	Fast claims before an external call; low-stakes dedupe
Stream processor state (Flink)	Yes, with the checkpoint	Local disk, checkpoint size	Dedupe inside a streaming job
Broker dedupe (SQS FIFO, Kafka producer)	Covers the broker’s records only	Nothing extra	Removing producer-retry duplicates, as one layer of several

Response to a recognized repeat	Wins when
Replay the stored first response	The caller needs the original result: a charge id, a decline code
“Already exists”, with the resource	Creates where the response is just the resource
409 while the first copy runs	Concurrent repeats; the caller retries later and then gets a replay
Silent success (ack, 200, no body)	Message consumers and webhooks: the sender only needs to stop

Never answer a recognized repeat with an error the caller treats as permanent. The repeat was a symptom of the network, not a mistake.

How it fails

Concurrent duplicate in flight. Two copies arrive within milliseconds: a balancer retry, a double tap, a redelivery while the first consumer is still working. Code that checks and then acts lets both through:

Check then act, and insert first

Scroll or pinch to zoom, drag to pan

• Color key
• Database
• Service

A unique index, SET NX, or a conditional write makes the claim and the check one step. The claim needs an in-flight state with a lease, so a copy that dies mid-work does not block its key forever.

Partial success, then a retry with a different key. The first attempt charged the card. The app crashed before saving its key. The restarted app pays again, under a new key. Every server guard works, and the buyer pays twice. Persist the key on the device before the first send. And add a business rule downstream: a unique constraint that allows one captured payment per order.

Expired dedupe window. A webhook retried on day 3 or a replay from last week finds no key and runs again. If caught repeats are near the TTL, uncaught ones are already past it.

Key collision. Two different operations share a key, and the second silently gets the first’s result. Causes: timestamps or weak randomness, keys not scoped by caller, content hashes, derivations that drop a field. Fingerprint the request with the key and refuse a mismatch with 422, so a silent loss becomes a visible bug.

A duplicate born upstream. Say a relay wraps each publish in a fresh event id. Or a producer mints the message id at send time, instead of storing it with the outbox row. Then a repeat looks new to every hop after it. Every hop must carry the id it was given.

Failure	Blast radius	Detect	Fix
Concurrent duplicate in flight	One operation, doubled	Two effects with the same key a few ms apart	Insert the key first; in-flight state with a lease
Retry with a new key	One customer each, but costly	Two successful payments per order in reconciliation	Key persisted on the client; unique business key downstream
Expired dedupe window	Every late copy	Age of caught repeats near the TTL; reconciliation diffs	Keys that live with the entity; TTL past the slowest path
Key collision	Silent loss of a real operation	422 fingerprint mismatches; missing effects	Random keys scoped by caller; fingerprint the request
Record not atomic with effect	Duplicates or lost work after crashes	Effects with no key, or keys with no effect	One transaction; intent first for external effects
Duplicate born upstream	Every hop after it	Two ids for one intent in traces	Carry ids, never mint new ones on republish
Zombie worker	One job, written twice or stale	Writes with an older lease token	Fencing tokens checked by the store
Zero rows reported as failure	Retries, DLQ noise, pages	DLQ messages whose effect already exists	Distinguish “already done” from “conflict”

Go deeperRetries that cause the outage1 min

Idempotency makes retries safe for data, not for load. Say each of four layers makes up to three attempts. Then one request can become 3⁴ = 81 attempts at the bottom, during an incident. Retry at one layer, usually the one nearest the caller. Use exponential backoff with jitter. And cap retries with a budget of about 10% of normal traffic, as the networking page explains.

Since 1.9.13, nginx does not pass a non-idempotent request (POST, LOCK, PATCH) to the next upstream once it was sent to one. The exception is when proxy_next_upstream includes non_idempotent. That is the right default. A service mesh retry policy is set per route, and needs the same care for POSTs.

At scale

The cost of remembering. Dedupe state grows with rate × window, so the window decides the bill:

Store	Calculation	Size
API keys, 3,000 writes a second, 24 h	3,000 × 86,400 = 259 M keys × ~200 B (row and index)	about 52 GB
Same, storing a 1 KB response each	259 M × ~1.2 KB	about 310 GB
Same keys kept 7 days	1.8 B keys × ~200 B	about 360 GB
Consumer inbox, 20,000 events a second, 7 days	20,000 × 604,800 = 12.1 B ids × ~40 B	about 480 GB, before index
Per-entity sequence, 50 M entities	50 M × ~16 B	about 800 MB, forever

The row sizes are assumptions. The shape is not: a per-operation record costs rate × window, a per-entity version costs the number of entities.

At 10x, the dedupe table becomes one of the busiest tables in the system: one insert per operation and one delete per expiry. Partition it by time (an hour or a day per partition) and drop whole partitions instead of deleting rows, which avoids vacuum or compaction storms. Shard it the same way as the data it protects, so the key and the effect stay in one transaction on one shard.

At 100x, and across regions, a key checked in one region is unknown in another. So pin each intent to a home region: the region that owns the account or the order. Then every retry of it lands where its key lives. A global dedupe check on every write costs a cross-region round trip of 60 to 150 ms, which is rarely acceptable. Streaming pipelines move dedupe into partitioned state, keyed by the same key as the data, so the check is local.

What to monitor:

• Repeats caught per hop per minute (replays, 409s, inbox hits, “already done” transitions). A jump means something upstream started retrying: a deploy, a slow dependency, a broken client.
• Age of a repeat when caught, as a histogram. Repeats near the TTL predict repeats past it.
• Fingerprint mismatches (422). A rising rate means a client reuses keys wrongly.
• Unknown outcomes: calls out that timed out and are waiting for a lookup, and their age.
• Reconciliation: effects per intent from the source of truth (two charges per order, two ledger lines per payment), compared daily. It is the only check that catches what every guard missed.

In the interview

Interviewer asksA user was charged twice. Walk me through where the duplicate could have come from.

Hop by hop. Maybe the app retried with a new key, perhaps after a restart. A balancer resent the POST. The relay published twice. Or the queue redelivered after the worker charged and crashed before acking. Or the worker retried the provider call without a key. Each has a guard. One key per intent, persisted on the client. A unique key, inserted first, in the same transaction as the payment. An inbox for messages. And a key derived from the payment id, sent to the provider. The crash after the charge is the one only the provider key fixes, because our own transaction rolled back. Then I would check reconciliation. It compares charges per order, and would have flagged it.

Interviewer asksHow long do you keep idempotency keys, and what happens after?

Longer than the slowest path a repeat can take. Client retries take seconds. An offline mobile queue takes hours or days. Stripe retries webhooks for up to three days. And a Kafka replay reaches back to the topic’s retention, seven days by default. So 24 hours covers online API retries, while consumers need days, sized to retention. After expiry, a repeat runs again. So where I can, I use keys that live with the entity: a unique constraint on order id and step, or a version per entity. Those never expire, and cost far less than a row per message.

Interviewer asksWe use Kafka with exactly-once enabled. Why do we still need idempotent consumers?

Kafka’s exactly-once covers Kafka. The idempotent producer stops a retried send from writing the record twice. And transactions commit a processor’s output records and its input offsets together. But a consumer that writes to Postgres, or calls a payment API, does that work outside the transaction. So a crash after the effect, and before the offset commit, replays it. And a relay that republishes after a crash writes a new record, carrying the same event id. So the consumer still dedupes by event id, or uses conditional writes, in the same transaction as its effect.

Common mistake

Generating a new idempotency key for each attempt

A key made inside the retry loop, or on each app launch, identifies the attempt, not the intent. Every server-side guard then sees each retry as new. The key is made once, when the user starts the operation, persisted with it, and reused on every retry until the intent finishes.

Common mistake

"The consumer dedupes, so the pipeline is exactly-once"

Dedupe at one hop only recognizes repeats of ids it has seen. A duplicate created upstream with a new id looks new to every hop after it. (Think of a retry with a new key, or a relay that mints new event ids.) And an external effect performed before the dedupe record commits is outside it. Exactly-once effects need a guard at each hop where repeats start, and keys that travel unchanged.

Mid-level covers

Adds an idempotency key to non-idempotent writes, and stores it with a unique constraint. Makes consumers dedupe by message id. Says exactly-once delivery is not possible, while effects can be made effectively once.

Senior adds

Walks every hop where repeats start, and names the guard for each. Prefers conditional writes and business keys over dedupe tables. Sizes the window to the slowest repeat path. Derives keys from one client intent, and passes them to providers. Picks a failure direction for effects that cannot be idempotent. States precisely what each exactly-once claim covers. Backs it all with reconciliation.

Seen in

• Payment system: the whole chain, from client key and provider attempt ids to compare-and-set states and a webhook inbox.
• Notification system: derived recipient ids, leased claims before sending, collapse ids for the unavoidable duplicate.
• Job scheduler: run ids from the fire time, fencing tokens, and the run id as the handler’s key.
• Ad click aggregator: dedupe by impression id and upserts of absolute totals that keep the larger value.
• Ticketmaster and hotel reservation: a key per hop, and conditional transitions that turn a late payment into a void.
• Leaderboard: ZINCRBY replaced by absolute totals with ZADD GT.
• Message queue: the idempotent producer and transactions, and where they stop.
• Stock exchange, chat and the collaborative editor: client ids plus sequence numbers (clOrdId, client message id, (clientId, seq)) so resends apply once.
• Ride hailing: a key on Request plus one active trip per rider, and the trip id as the payment’s key.
• File sync: blocks named by their hash, and commits conditional on the parent revision.

Recap

The goal. One intent, one effect, whatever is repeated on the way: at-least-once delivery plus an idempotent step at every hop where repeats start.

Three tools, in order. Write states, not deltas. Make transitions conditional on current state or version, and treat zero rows on a repeat as “already done”. Remember keys, inserted first, in the same transaction as the effect, with an in-flight state. Fence actors whose lease ended.

The window. A dedupe record must outlive the slowest repeat: webhooks for days, log replays for the retention period. Keys that live with the entity, or a version per entity, have no window.

Keys. One per user intent, made where the intent starts and persisted. Derive child keys per hop and per attempt. Never key actions by content, never mint new ids on republish.

Other systems. Send them your key and know their retention. After it, look up by reference before retrying.

Non-idempotent effects. Choose lose or repeat per message type, then shrink the duplicate: leased claims, collapse ids, idempotent content, caps.

Exactly once. Every product’s claim stops at its own boundary. State the guarantee as an observable property and reconcile against it.

Further reading

• On this site: API design, for the Idempotency-Key header and key store. Message queues, for delivery semantics and the outbox. Distributed transactions, for sagas and the inbox. And consensus, for leases and fencing tokens.
• Designing robust and predictable APIs with idempotency, Stripe: keys, retries with backoff, and how a payment API uses them.
• Exactly-once semantics are possible: here’s how Kafka does it, Confluent: the idempotent producer and transactions, from the people who built them.
• CronJob, Kubernetes documentation: why a scheduled job may run twice or not at all, and must be idempotent.