Distributed transactions and sagas

The problem it solves

A shop’s checkout touches three services, each with its own database: inventory reserves the item, payments charges the card, shipping books a courier. In a monolith with one Postgres database this is one transaction: all three rows change, or none do.

Split across services, checkout becomes three separate commits. A deploy kills the checkout process after the charge commits and before shipping is booked. The customer has paid, the item is reserved, and nothing will ever ship it. No error was logged, because the process that would have logged it is gone.

How often? As a planning figure, assume 1 checkout in 100,000 is interrupted between two writes, by deploys, OOM kills, node failures or timeouts. A shop doing 500 checkouts a second then gets 500 × 86,400 / 100,000 ≈ 430 broken orders a day. Each one is a support ticket, or a chargeback.

This is the dual write problem. Sometimes one operation must change two systems that do not share a transaction. Then any failure between the two writes leaves them disagreeing. It appears across services. It appears across shards of one database, like a transfer between accounts on two shards. And it appears between a database and anything else: a Kafka message, a search index, a payment provider.

A dual write: one crash, two systems that disagree forever

Scroll or pinch to zoom, drag to pan

• Color key
• Database
• Service

There are two families of answers. Atomic commit (two-phase commit) makes several databases act as one, and nobody sees the in-between. Sagas let each step commit on its own and undo finished steps with compensations, with the in-between visible for a while.

What makes this hard

• No protocol makes a remote call “exactly once”. A call can succeed, fail, or time out with no answer, and the third case is the one that breaks designs.
• Atomic commit across machines blocks when the coordinator dies at the wrong moment. Avoiding the blocking means giving up isolation, and the anomalies that follow have to be designed away.
• Undoing is a business action, not a rollback. A refund is not the absence of a charge: the customer saw both.

How it works

Step 1: One database, one transaction

First, check whether the data can live in one database. A local transaction gives atomicity and isolation for free. One PostgreSQL primary runs about 10,000 write transactions a second (see databases). That covers most businesses.

Service boundaries are a choice. Data that must always change together belongs in one service and database. The payment system keeps payment state, ledger lines and the outgoing event in one Postgres for this reason. What forces data apart is separate teams, one database’s write limit, shards, or data in someone else’s system (a card network).

Step 2: Calls that might have happened

Every remote call has three outcomes, not two:

Outcome	What you know	Safe next step
Success	It happened	Continue
Failure	It did not happen (a 4xx, a declined card)	Stop or undo the earlier steps
Timeout	Nothing. The request or the reply may be lost	Ask again with the same key, or check status

Treating a timeout as a failure is wrong whenever the work actually happened. Treating it as success is wrong whenever it did not.

The fix is to make every step safe to repeat. Each request carries an idempotency key. It’s derived from the business operation (order 42, step 2), not generated per attempt. The receiver stores its result under that key, in the same local transaction as the effect. A repeated key returns the stored result, instead of acting again. Stripe, for example, returns the first result for a repeated Idempotency-Key, and keeps keys for at least 24 hours. Every protocol below retries. So every protocol below depends on this.

Step 3: Two-phase commit

Two-phase commit (2PC) makes several databases commit atomically. One process is the coordinator (a transaction manager). Each database is a participant.

1. Prepare. The coordinator asks every participant to prepare. A participant that can commit writes all the transaction’s changes and a prepare record to disk, keeps its locks, and votes yes. From then on it may not abort on its own. A participant that cannot commit votes no and rolls back.
2. Commit. If every vote is yes, the coordinator writes commit to its own log and forces it to disk. That write is the commit point: the transaction is committed whatever crashes next. It then tells every participant to commit, retrying until each acknowledges. If any vote was no or missing, it tells everyone to abort.

Two-phase commit: two round trips and three forced log writes on the critical path

Scroll or pinch to zoom, drag to pan

• Color key
• Database
• Service

Most implementations use presumed abort. The coordinator forces no log record for an abort, and needs no acknowledgements for it. On restart, or when an in-doubt participant asks, it answers “abort” for any transaction it has no commit record for. That is safe, because no participant can have been told to commit before that record was on disk. Only commits pay for a forced write.

The blocking problem. Suppose every participant has voted yes and the coordinator dies before telling anyone the outcome. Each prepared participant is now in doubt. It cannot commit, because another participant may have voted no. It cannot abort, because the coordinator may have written commit and told someone else before dying. So it waits, holding its row locks, until the coordinator comes back. Every other transaction that needs those rows waits behind it. If the coordinator’s disk is gone, a human decides.

The coordinator dies after the votes: participants in doubt, locks held, others queue

Scroll or pinch to zoom, drag to pan

• Color key
• Service
• Database

Participants can ask each other. If any one voted no, or already heard the decision, all can finish. But when all voted yes and none heard, nobody can decide safely. 2PC cannot fix this while the coordinator is a single machine. Three-phase commit adds a round. But it’s only non-blocking with bounded message delays and no partitions. So production systems do not use it.

The code below is the whole protocol with a crash in the worst place. The participants’ only correct move is to wait.

JParticipant.javaPycoordinator.pyC#Participant.csGocoordinator.goTScoordinator.ts

JavaPythonC#GoTypeScript

JavaPythonC#GoTypeScript

import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;

// Stands in for the coordinator process dying.
class Crash extends RuntimeException {
    Crash(String message) { super(message); }
}

// One database in the transaction, such as a Postgres shard using PREPARE TRANSACTION.
class Participant {
    final String name;
    final boolean votesYes;
    String state = "working"; // working -> prepared -> committed | aborted
    boolean locksHeld = true; // row locks taken while the transaction did its writes

    Participant(String name, boolean votesYes) {
        this.name = name;
        this.votesYes = votesYes;
    }

    Participant(String name) { this(name, true); }

    boolean prepare(String tx) {
        if (!votesYes) {
            finish("aborted"); // a "no" may abort at once: the transaction can never commit
            return false;
        }
        // Force the writes and a prepare record to disk. From here on the participant has
        // promised to commit if told, even after its own crash, so it keeps every lock.
        state = "prepared";
        return true;
    }

    void finish(String outcome) {
        state = outcome;
        locksHeld = false;
    }

    String whenCoordinatorSilent() {
        if (state.equals("prepared")) {
            // It cannot commit: another participant may have voted no. It cannot abort: it
            // promised, and the coordinator may already have told someone else to commit.
            return "in doubt, waiting, locks held";
        }
        return "free to decide alone, it never promised";
    }
}

class Coordinator {
    // Durable. Under presumed abort only commit decisions need a record.
    private final Map<String, String> log;

    Coordinator(Map<String, String> log) { this.log = log; }

    String run(String tx, List<Participant> parts, boolean crashBeforeDecision) {
        boolean allYes = true;
        for (Participant p : parts) { // phase 1, in parallel in real systems
            allYes &= p.prepare(tx);
        }
        if (crashBeforeDecision) throw new Crash("coordinator died after the votes, before deciding");
        String decision = allYes ? "commit" : "abort";
        if (decision.equals("commit")) {
            log.put(tx, "commit"); // the commit point: on disk before any participant hears
        }
        tell(parts, decision); // phase 2, retried until every participant acknowledges
        return decision;
    }

    String recover(String tx, List<Participant> parts) {
        // Presumed abort: no record means no participant can have been told to commit.
        String decision = log.getOrDefault(tx, "abort");
        tell(parts, decision);
        return decision;
    }

    private static void tell(List<Participant> parts, String decision) {
        for (Participant p : parts) {
            if (p.state.equals("prepared")) { // a "no" voter already aborted
                p.finish(decision.equals("commit") ? "committed" : "aborted");
            }
        }
    }
}

class Main {
    static String show(List<Participant> parts) {
        return parts.stream().map(p -> p.name + " " + p.state).collect(Collectors.joining(", "));
    }

    public static void main(String[] args) {
        Map<String, String> log = new HashMap<>();

        List<Participant> t1 = List.of(new Participant("orders"), new Participant("stock"), new Participant("ledger"));
        System.out.println("tx-1: " + new Coordinator(log).run("tx-1", t1, false) + " | " + show(t1));

        List<Participant> t2 = List.of(new Participant("orders"), new Participant("stock"), new Participant("ledger"));
        try {
            new Coordinator(log).run("tx-2", t2, true);
        } catch (Crash e) {
            System.out.println(e.getMessage());
        }
        for (Participant p : t2) System.out.println("  " + p.name + ": " + p.whenCoordinatorSilent());
        // Nothing moves until a coordinator with the same log comes back, however long that takes.
        System.out.println("tx-2 after restart: " + new Coordinator(log).recover("tx-2", t2) + " | " + show(t2));
        // tx-1: commit | orders committed, stock committed, ledger committed
        // coordinator died after the votes, before deciding
        //   orders: in doubt, waiting, locks held
        //   stock: in doubt, waiting, locks held
        //   ledger: in doubt, waiting, locks held
        // tx-2 after restart: abort | orders aborted, stock aborted, ledger aborted
    }
}

XA is the X/Open standard interface for this. A transaction manager (in Java, a JTA implementation) coordinates XA-capable databases and brokers. New designs rarely use it across services. Locks span the whole protocol. Every participant must be up at once. And card networks and REST APIs cannot prepare.

Go deeperXA in practice1 min

Postgres supports it with PREPARE TRANSACTION (off by default: max_prepared_transactions is 0), MySQL with XA PREPARE. The costs in detail:

• Lock time. Locks are held across two round trips and several fsyncs. That’s a few milliseconds on one network, and two 70 ms round trips or more across regions.
• Availability multiplies. The transaction needs every participant and the coordinator up at once. Three services at 99.9% give 99.7% for the whole.
• Most participants cannot prepare. A card network, a courier’s REST API, an email provider and most managed queues have no prepare phase. You cannot put an HTTP call inside 2PC.
• Operational traps. A prepared transaction orphaned by a lost coordinator survives restarts, and holds its locks. In Postgres, it stops VACUUM from removing dead rows, until someone finds it in pg_prepared_xacts.

Where 2PC is alive and well: inside one distributed database. Spanner, CockroachDB, YugabyteDB and TiDB run 2PC for every multi-shard transaction. Here’s what removes the blocking problem. Each participant is a Paxos or Raft group of replicas. And the decision is written to a replicated record. (Spanner’s coordinator is itself a Paxos group. CockroachDB keeps a transaction record in a Raft range.)

So one machine failing means a leader election of a few seconds, not a decision lost on one disk. Every participant learns the outcome. Kafka’s transactions work the same way, with the coordinator’s log in a replicated topic. 2PC is a fine tool inside one system, whose participants are replicated and run by one team. It’s the wrong tool across independently owned services.

Go deeperCommit latency inside distributed SQL, and Percolator2 min

Distributed databases shave the cost in different ways. CockroachDB’s parallel commits write the transaction record as “staging”, in parallel with the last writes. So a commit costs about one round of consensus, instead of two. Any reader that finds a staging record can tell whether all writes landed, and finish the commit itself. Spanner adds commit wait. The coordinator holds the commit until TrueTime’s uncertainty (single-digit milliseconds) has passed. That gives external consistency (see consistency models).

Google’s Percolator (2010) built 2PC on top of Bigtable, without a coordinator process. The client writes locks into the rows themselves. One of them is the primary lock. Committing the primary is the commit point. A reader that finds a stale secondary lock checks the primary, and rolls the transaction forward or back. TiDB’s transaction model descends from it. The pattern applies widely. Put the coordinator’s decision in a replicated record that anyone can read. Then nobody is ever stuck in doubt.

Tradeoff2PC inside one replicated database, never across services

We gain

Real atomicity and isolation across shards, and the application code stays an ordinary transaction.

We give up

Consensus rounds and held locks on every multi-shard write. Across services it blocks on a dead coordinator and cannot include any system without a prepare phase.

Step 4: Sagas

A saga (Garcia-Molina and Salem, 1987) was first written for long-lived transactions in one database. It splits the work into a sequence of local transactions. Each one commits on its own, in one service. Every step that may need undoing has a compensating transaction. That’s a new action that reverses it in business terms: release the reservation, refund the charge, cancel the courier. If step k fails, the saga runs the compensations of steps k−1 down to 1, in reverse order.

Step	Service	Action	Compensation
1	Inventory	Reserve 1 unit	Release the reservation
2	Payments	Charge 40 dollars	Refund 40 dollars
3	Shipping	Book a courier	Cancel the booking

The order sits in PENDING while the saga runs, and ends as APPROVED or REJECTED. No locks span steps, so nothing blocks when a service dies. The price is that others can see the in-between: the card is charged before shipping has said yes. A saga is atomic in the end (all steps, or all compensated) but not isolated. Step 6 covers what that costs.

An orchestrated saga: shipping fails, finished steps are compensated in reverse

Scroll or pinch to zoom, drag to pan

• Color key
• Service
• Database

Four rules make a saga correct rather than merely optimistic:

1. The saga’s state is durable. The orchestrator writes its progress (which step, which direction) to its database, before acting on it. A restarted orchestrator continues from there. Progress held in memory is the classic lost compensation.
2. Every step and every compensation is idempotent. Each carries a key derived from the saga id and step number. So after a crash or a timeout, the orchestrator simply sends it again.
3. Compensations always succeed eventually. There is no “give up” branch: retry with backoff until it lands, and page a human past a threshold. So a compensation must be something the business can always do: a refund, not “un-send an email”.
4. A compensation tolerates arriving first. After a timeout, the orchestrator compensates a step whose outcome it does not know. The participant records the cancellation, even with nothing to undo. And it refuses the forward request if it shows up late. So a delayed charge cannot land after its own refund.

Pivot transactions. Not every step can be compensated. Once the courier has collected the parcel, there is no undo. So order the saga in three parts:

1. Compensatable steps first.
2. Then the pivot: the go or no-go step, after which the saga can only go forward.
3. Then retriable steps, which succeed if retried long enough (the confirmation email, the search index).

Steps likely to fail go early.

JOrchestrator.javaPyorchestrator.pyC#Orchestrator.csGoorchestrator.goTSorchestrator.ts

JavaPythonC#GoTypeScript

JavaPythonC#GoTypeScript

import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.function.Function;
import java.util.stream.Collectors;

// The call never reached the service. Safe to retry.
class Unavailable extends RuntimeException {
    Unavailable(String name) { super(name); }
}

// The service did the work but the reply never arrived. A timeout looks exactly like this.
class ReplyLost extends RuntimeException {
    ReplyLost(String name) { super(name); }
}

// Stands in for the orchestrator process dying.
class Crash extends RuntimeException {
    Crash(String message) { super(message); }
}

// One participant with its own database. Every request carries a key built from the saga id and
// the step number. The service stores its answer under that key in the same local transaction as
// the effect, so a retry gets the first answer instead of a second effect.
class Service {
    final String name;
    final boolean declines;
    boolean loseReplyOnce;
    int undoDownFor; // the next N compensation calls fail
    final Map<String, String> results = new HashMap<>(); // key -> ok | declined | cancelled
    int effects = 0; // reservations, charges or bookings currently in force
    int calls = 0;

    Service(String name, boolean declines, boolean loseReplyOnce, int undoDownFor) {
        this.name = name;
        this.declines = declines;
        this.loseReplyOnce = loseReplyOnce;
        this.undoDownFor = undoDownFor;
    }

    static Service ok(String name) { return new Service(name, false, false, 0); }

    String act(String key) {
        calls++;
        if (!results.containsKey(key)) { // BEGIN
            if (declines) {
                results.put(key, "declined"); // a business "no": nothing to undo
            } else {
                effects++;
                results.put(key, "ok");
            }
        }
        // COMMIT. A key that is already "cancelled" falls through: a request that arrives after
        // its own compensation is refused, so it cannot leave an orphaned charge.
        if (loseReplyOnce) {
            loseReplyOnce = false;
            throw new ReplyLost(name);
        }
        return results.get(key);
    }

    String compensate(String key) {
        if (undoDownFor > 0) {
            undoDownFor--;
            throw new Unavailable(name);
        }
        if ("ok".equals(results.get(key))) { // BEGIN
            effects--; // a refund or a release: a new action, not a rollback
        }
        results.put(key, "cancelled"); // written even if the action never arrived
        return "cancelled"; // COMMIT. A second compensation with the same key changes nothing.
    }
}

// Runs one saga: each step forward in order, and on failure every started step's compensation in
// reverse. Its progress lives in a durable table (log), written before it moves on, so a
// restarted orchestrator resumes where the dead one stopped.
class Orchestrator {
    static final class Row {
        int next = 0;
        String status = "running";
    }

    private final Map<String, Row> log;
    private final List<Service> services;
    private final Integer crashAfterStep;

    Orchestrator(Map<String, Row> log, List<Service> services, Integer crashAfterStep) {
        this.log = log;
        this.services = services;
        this.crashAfterStep = crashAfterStep;
    }

    String run(String sagaId) {
        Row row = log.computeIfAbsent(sagaId, id -> new Row());
        while (row.status.equals("running") && row.next < services.size()) {
            int k = row.next;
            String reply = tryCall(services.get(k)::act, sagaId + ":" + k, 3);
            if (crashAfterStep != null && crashAfterStep == k) {
                throw new Crash("orchestrator died after step " + k + ", before saving the reply");
            }
            if (reply.equals("ok")) {
                row.next = k + 1; // saved before the next step is sent
            } else {
                row.status = "compensating"; // declined, or still unknown after 3 tries
            }
        }
        if (row.status.equals("compensating")) {
            // Reverse order, and including step `next`: if its outcome is unknown it may have
            // happened, and compensating something that never happened is a harmless no-op.
            for (int k = row.next; k >= 0; k--) {
                untilDone(services.get(k)::compensate, sagaId + ":" + k);
            }
            row.status = "compensated";
        } else if (row.status.equals("running")) {
            row.status = "completed";
        }
        return row.status;
    }

    private static String tryCall(Function<String, String> call, String key, int attempts) {
        for (int i = 0; i < attempts; i++) {
            try {
                return call.apply(key); // the same key every time makes the retry safe
            } catch (Unavailable | ReplyLost e) {
                // real code sleeps here: exponential backoff with jitter
            }
        }
        return "unknown";
    }

    private static void untilDone(Function<String, String> call, String key) {
        // A compensation has no "give up" branch: the saga is stuck half done until it lands.
        // Real code backs off, alerts after a threshold and parks the saga for a human.
        while (true) {
            try {
                call.apply(key);
                return;
            } catch (Unavailable e) {
                // retry
            }
        }
    }
}

class Main {
    static String show(List<Service> s) {
        return s.stream().map(x -> x.name + " " + x.effects).collect(Collectors.joining(", "));
    }

    public static void main(String[] args) {
        Map<String, Orchestrator.Row> log = new HashMap<>();

        // Saga 1: shipping has no courier. The refund fails twice before it lands.
        List<Service> s1 = List.of(Service.ok("inventory"), new Service("payments", false, false, 2),
                new Service("shipping", true, false, 0));
        System.out.println("saga-1: " + new Orchestrator(log, s1, null).run("saga-1") + " | " + show(s1));

        // Saga 2: the charge's reply is lost once, then the orchestrator dies after the charge.
        List<Service> s2 = List.of(Service.ok("inventory"), new Service("payments", false, true, 0),
                Service.ok("shipping"));
        try {
            new Orchestrator(log, s2, 1).run("saga-2");
        } catch (Crash e) {
            System.out.println(e.getMessage());
        }
        // A new process, the same durable log.
        System.out.println("saga-2: " + new Orchestrator(log, s2, null).run("saga-2") + " | " + show(s2));
        System.out.println("payment calls: " + s2.get(1).calls + " charges in force: " + s2.get(1).effects);
        // saga-1: compensated | inventory 0, payments 0, shipping 0
        // orchestrator died after step 1, before saving the reply
        // saga-2: completed | inventory 1, payments 1, shipping 1
        // payment calls: 3 charges in force: 1
    }
}

Try it. Run the checkout as a saga, then as 2PC. Make a step fail, or lose its reply. Crash the coordinator in the middle. And turn idempotency keys off, to see where double charges come from.

SagaTwo-phase commit

1. Reserve inventory

SucceedsFailsReply lost

2. Charge card

SucceedsFailsReply lost

3. Book shipping

SucceedsFailsReply lost

Orchestrator crashes

NeverIn step 1In step 2In step 3

Idempotency keys Compensations fail twice first

Run checkoutRestart orchestratorReset

Orchestratoridlesaga log: empty

Inventoryidle 

Paymentsidle 

Shippingidle 

Locks held while the coordinator is down

0 s

Checkouts waiting on those rows

0

Pick an outcome for each step and where the coordinator crashes, then run the checkout.

A crashed orchestrator locks nothing, but the half-done order is visible until it restarts. A crashed 2PC coordinator shows nothing half-done, but freezes every row the transaction touched.

Step 5: Orchestration or choreography

A saga needs something to drive it. There are two shapes.

Orchestration. One component owns the saga’s state machine, sends a command to each participant and waits for the reply. The flow lives in one place. Workflow engines such as Temporal, AWS Step Functions and Camunda are orchestrators with durable state, timers and retries built in.

Choreography. No one is in charge. Each service reacts to the others’ events: Inventory hears OrderCreated, reserves and publishes StockReserved, and a failure event makes the earlier services compensate. It needs no extra component.

Choreography: the flow exists only in who listens to what

Scroll or pinch to zoom, drag to pan

• Color key
• Service

Choreography’s cost grows with the saga. With eight steps no file says what checkout does, and “where is order 42 stuck?” needs a trace across every service. A common rule: choreography for two or three steps, orchestration beyond that or once there are branches and timeouts.

Step 6: Living without isolation

Each step commits on its own. So concurrent sagas see each other’s intermediate state. And the anomalies that isolation levels normally prevent come back (see consistency models):

• Dirty read. A wallet saga credits 100 dollars from a pending top-up, and a payment saga reads the new balance and spends it. The top-up’s card is then declined and the credit is compensated. The wallet is overdrawn.
• Lost update. Saga A reads order 42 in step 1. Saga B cancels order 42. Saga A’s last step writes the order back as APPROVED, and the cancellation is gone.
• Non-repeatable read. A saga reads a price in step 1, and it changes before the charge in step 3.

The countermeasures are design patterns, not database settings. The four that matter most:

• A semantic lock: a PENDING state that others respect.
• Commutative updates: subtract 40, instead of setting the balance to 60.
• A pessimistic view: order steps so risky effects come last. Credit the wallet only after the top-up is captured.
• Reread value: recheck the price at the charge step.

Go deeperAll six countermeasures1 min

Countermeasure	How it works	Example
Semantic lock	A step sets a pending state that other requests respect, cleared when the saga ends	Order is PENDING; a second cancel or edit is refused until then
Commutative update	Express changes so the order of application does not matter	balance = balance - 40, not balance = 60
Pessimistic view	Order the steps to minimize the damage of a dirty read: risky effects last	Credit the wallet only after the top-up’s card payment is captured
Reread value	Before updating, reread and check the value has not changed; abort and restart the saga if so	Recheck the price at the charge step, compare with the quoted one
Version file	Record operations as they arrive and apply them in a correct order	A cancel that arrives before its create is held, then applied
By value	Pick the mechanism per request by risk	Small orders run as a saga, large transfers go through one ledger

Almost every real saga uses a semantic lock. The order’s PENDING state refuses a second cancel, until the saga ends. In the hotel and ticket designs, a HELD reservation counts against inventory while the card is authorized. Unlike a database lock, it should expire, so a dead saga cannot block forever. And the saga must treat “my hold expired” as a normal outcome.

Step 7: Reliable messages between steps

Each saga step is itself a small dual write: change my data, and tell the next step.

Sending: the transactional outbox. The business change and an outbox row commit in one local transaction. A relay publishes the rows afterwards, by polling, or by change data capture (Debezium reading the write-ahead log). A crash makes the event late or repeated, never lost. (The code is in message queues, step 7.) The orchestrator uses it too. “Step 1 done, send charge” is one transaction on the saga table, plus an outbox row with the command.

Receiving: the inbox, or idempotent consumer. Every message can arrive twice. So the consumer records the message’s key in the same transaction as its effect, under a unique constraint. And it skips keys it has seen.

One saga step, end to end: inbox, effect and outbox in one local transaction

Scroll or pinch to zoom, drag to pan

• Color key
• Queue or stream
• Service

Together, they give an exactly-once effect per hop. Delivery is at least once, and the duplicates do nothing. Exactly-once delivery across a network does not exist. Kafka’s exactly-once covers reads and writes inside Kafka, not your database, or a card network.

Step 8: Try-Confirm-Cancel

TCC turns each step into a reservation. Every participant offers three operations:

• Try checks the business rules and reserves the resource, with an expiry. Nothing is consumed and the reservation is invisible as a finished effect: the stock is “held”, the funds are “frozen”.
• Confirm turns the reservation into the real effect. It must succeed if Try succeeded. It’s retried until it does.
• Cancel releases the reservation. Also retried until it succeeds.

The coordinator runs Try on everyone, then Confirm on everyone if all Tries succeeded, otherwise Cancel. It looks like 2PC, but each call is a committed local transaction, so no database locks are held between phases. The reservation is a semantic lock with a timeout: if the coordinator disappears, reservations expire and nobody blocks forever.

Card payments work this way. Authorize is Try (the issuer holds the funds, commonly for up to about 7 days for online payments), capture is Confirm, and void is Cancel.

Confirm and Cancel are idempotent, by transaction id. Every participant must handle three cases:

• An empty cancel: Cancel arrives for a Try that never landed. Record it, and succeed.
• A hanging try: Try arrives after its Cancel. Refuse it, because the cancellation record exists.
• An expired reservation: Confirm arrives after the hold lapsed, and the stock sold. Fail, and the coordinator cancels the rest and tells the client.

The last case is why “Confirm always succeeds” is a goal, not a guarantee. Set the expiry well above the coordinator’s worst-case delay. And treat expiry as an ordinary outcome.

TradeoffTCC where the domain already has holds

We gain

Better isolation than a plain saga: others see a reservation, not a finished effect that may be compensated. Nothing blocks on a dead coordinator, because reservations expire.

We give up

Every participant implements three operations, a reservation state and the edge cases. Reserved resources are unavailable to others until confirmed or expired.

Options and tradeoffs

Atomic commit, saga or TCC

Approach	Atomic	Isolated	On a coordinator crash	Can include third-party APIs	Wins when
One local transaction	Yes	Yes	Not applicable	No	The data can live in one database. Always check this first
2PC / XA across databases	Yes	Yes	Participants block holding locks	No	A few databases you run, same data center, low volume, legacy Java EE estates
Distributed SQL (2PC over consensus)	Yes	Yes, up to serializable	Brief pause for leader election	No	Cross-shard transactions are the normal case and you want SQL to handle them
Saga (orchestrated or choreographed)	Eventually, via compensations	No: intermediate states visible	Nothing blocks; resumes from its log	Yes	Steps span services and external APIs; compensations exist for every early step
TCC	Eventually, via cancel	Partly: reservations, not finished effects	Reservations expire	Yes, if they offer holds	The domain has natural holds: seats, rooms, stock, card authorizations

TradeoffA saga by default across services, a local transaction wherever data can share a database

We gain

Each service keeps its own database and stays available when others are down. Third-party calls fit in as steps, and nothing waits on a lock held by a dead process.

We give up

No isolation, so semantic locks and step ordering must be designed. Every step needs a compensation and a key, and the saga a durable state machine and stuck-saga monitoring.

Orchestration or choreography

Option	Where the flow lives	Wins	Loses
Orchestration	One state machine, in code or an engine	One place to read, test and change; easy timeouts and “where is it now”	Another component to run; risk of it growing into a god service that owns logic
Choreography	Spread across every service’s subscriptions	No central component; services evolve independently	Hard to see and change the whole flow past 3 or 4 steps; event cycles; tracing needed
Hybrid	Orchestrator per business process, events between bounded contexts	Each team orchestrates its own flow, others react to its events	Two styles to understand

TradeoffOrchestrate checkout, publish events for everyone else

We gain

The steps that must succeed or be undone together live in one readable state machine. Downstream consumers (email, analytics, search) just subscribe.

We give up

The orchestrator is a service to run, with its own database and on-call. And it must stay sequencing only, or business rules leak out of the services that own them.

Making each hop exactly-once in effect

Technique	How it works	Wins when
Naturally idempotent operation	SET status = 'shipped', upserts of absolute values	The operation can be expressed as a state, not a delta
Unique constraint on the business key	UNIQUE (saga_id, step) on the payments table; a repeat fails the insert	The effect is one row you own. The cheapest correct option
Inbox table	Message id inserted in the same transaction as the effect	Many message types, effects spread over several tables
Downstream idempotency key	Send Idempotency-Key: o42-2 to the provider	The effect happens in someone else’s system; check their key retention

TradeoffA unique key in the same transaction as the effect

We gain

Duplicates become no-ops by construction, enforced by the database rather than by code that checks first and writes later.

We give up

Every message must carry a stable key, and dedupe records need a retention policy longer than the longest possible redelivery or replay.

How it fails

Failure	What breaks	How you detect it	Blast radius	Fix
Coordinator dies in doubt	2PC participants hold locks until it returns	Count and age of prepared transactions	Every transaction touching the locked rows	Replicate the coordinator’s log (consensus), or avoid cross-service 2PC; a runbook for manual resolution
Double charge	A retry after a timeout, or a recovery, repeats a step with no key, or with a new key per attempt	Two charges per order in reconciliation; customer disputes	One customer each, but costly and public	Keys derived from saga id and step, never per attempt; provider idempotency keys
Lost compensation	The orchestrator kept progress in memory and crashed between failure and compensating	Orders stuck in PENDING with a charge and no shipment	Each affected order	Durable saga state written before acting; compensations sent through the outbox
Stuck saga	A participant is down for hours, or a reply never comes and nothing times out	Saga age by state; oldest running saga over its SLA	Customers waiting; semantic locks held	A timeout per step that triggers retry or compensation; park and page after N attempts
Timeout treated as failure	The step actually succeeded; the saga compensates or, worse, the client tries again with a new order	Refunds without a matching failure; “late success” logs	Silent double work or a refund for a good order	Retry with the same key or query status before deciding; compensations that tolerate arriving first

Go deeperSeven more failures2 min

Failure	What breaks	How you detect it	Fix
Orphaned prepared transaction	A Postgres prepared transaction outlives its coordinator; holds locks and blocks VACUUM	pg_prepared_xacts rows older than a minute	Alert on age; resolve with COMMIT PREPARED or ROLLBACK PREPARED from the coordinator’s log
Heuristic decision	An XA participant tired of waiting commits or aborts on its own; others did the opposite	Heuristic-mixed exceptions from the transaction manager	Disable heuristics or reconcile by hand; one more reason to avoid XA across teams
Two orchestrators, one saga	A consumer rebalance or a slow pause lets two instances drive the same saga	Conflicting commands with the same saga id	A version column on the saga row, updated with compare-and-set; stale instance loses
Hanging try or late forward call	A delayed request lands after its own compensation and creates an orphan effect	Effects with no live saga in reconciliation	Record cancellations and refuse later forward calls with the same key
Dedupe window too short	Keys expire before a replay or a slow retry arrives, so a duplicate is processed	Duplicate effects after a consumer replay	Keep keys longer than broker retention plus the longest retry; or use a business unique key
Poison message	A command that always throws is retried forever and blocks its partition	Same message redelivered; consumer lag on one partition	Retry limit, then dead-letter queue, and tell the orchestrator the step failed so it compensates
Compensation that cannot run	The step was irreversible (email sent, parcel shipped) but came before a step that failed	Compensation errors with “not supported”	Reorder: pivot and irreversible steps last; replace undo with a corrective action (apology, credit)

Most of these share a root: something assumed a message arrives exactly once, or that a silent participant had failed. The last line of defense is a daily reconciliation that compares orders, charges and shipments and flags any saga that ended inconsistent.

At scale

What a saga costs. Each step is one local transaction (1 to 5 ms), plus a message hop each way when orchestrated. A hop through an outbox, Kafka and a consumer takes tens of milliseconds with CDC. With a polling relay, it takes up to the polling interval (100 ms to 1 s). A three-step orchestrated saga is six hops: about 100 to 300 ms with CDC. Checkout usually answers “order received” early, and finishes in the background.

What 2PC costs. Two round trips and three sequential forced log writes, with every lock held throughout: a few milliseconds in one data center. Across regions 70 ms apart it is at least 140 ms, so a hot row manages about 7 transactions a second.

Sizing the state. At 1,000 checkouts a second, the saga table gains 86.4 million rows a day. Assume about 500 bytes a row: that’s about 43 GB. Keeping dedupe keys for 7 days means about 1.8 billion keys. Partition both by day, and drop old partitions, instead of deleting rows.

Go deeperThe sizing arithmetic1 min

Item	Arithmetic	Result
Saga rows a day	1,000 × 86,400	86.4 million rows
Saga table growth	86.4 M × 500 bytes (assumed)	about 43 GB a day
Messages a second	1,000 × 3 steps × command + reply	6,000 a second
Dedupe keys kept 7 days	3,000 commands a second × 604,800 s	about 1.8 billion keys
Dedupe storage	1.8 B × 40 bytes (assumed)	about 73 GB

Finished sagas move to cheap storage after the refund window, often 30 to 90 days.

At 10x (10,000 sagas a second), the orchestrator becomes many stateless workers. Commands and replies are keyed by saga id in Kafka, so one saga’s messages stay in order, on one worker. The saga table is sharded the same way. Timeouts move to a durable timer: an indexed wake_at column, or the workflow engine’s timers. The participants, not the orchestrator, become the bottleneck.

At 100x, look at the shape again. Every step that crosses a service boundary costs two messages and a dedupe record. So steps that always happen together are cheaper merged into one local transaction. A flash sale on one SKU concentrates semantic locks on one row. So split the stock into buckets, and reservations spread over several rows.

What to monitor.

1. Saga count by state, and the age of the oldest saga in each non-final state. Page when the oldest running saga exceeds its SLA (for checkout, minutes).
2. Compensation rate per step, and compensations retried more than N times (page a human).
3. Outbox lag: age of the oldest unpublished row, per service.
4. Dead-letter queue depth, with the saga id of each message.
5. For 2PC: count and age of prepared or in-doubt transactions. Anything older than a minute is a page.
6. Daily reconciliation mismatches across orders, payments and shipments. The target is zero.

In the interview

Interviewer asksWhy not use two-phase commit across the order, payment and inventory services?

The card network and the courier API cannot prepare. So 2PC cannot include the steps that matter most. A coordinator that dies after the votes leaves participants in doubt. They hold locks that block every checkout on those rows. And every service must be up at once. I would use a saga: local transactions with idempotency keys, a compensation per early step, a durable orchestrator, and the order PENDING as a semantic lock. 2PC is fine inside Spanner or CockroachDB, where every participant is a replicated group.

Interviewer asksThe card was charged, then shipping says no courier is available. Walk me through what happens.

The saga table already says “charge done”. The orchestrator saves “compensating”, then sends the refund with key o42-2-undo and the stock release, each retried until it lands and applied once. The order moves to REJECTED. If the orchestrator crashes halfway, a new instance reads “compensating” and resends. The customer sees a charge and a refund, which is the isolation a saga gives up. To avoid that, authorize instead of charging, capture after shipping confirms, and void on failure: TCC with the card network’s own holds.

Interviewer asksThe call to the payment provider timed out. Do you retry, refund, or fail the order?

None of those blindly. A timeout means “unknown”. I retry with the same idempotency key, derived from the order and step. So the provider either returns the original result, or runs it once. If it stays unknown past the step’s deadline, I look the payment up by our reference. I void or refund it if it exists. And I record the cancellation, so a charge that lands later is refunded. Reconciliation against the provider’s report catches anything left.

Interviewer asksOrchestration or choreography for checkout?

Orchestration. Checkout has several steps, a pivot, timeouts and compensations. And support needs one place that says “order 42 is waiting on shipping, step 3, for 6 minutes”. With choreography, that state is scattered over five services. The orchestrator only sequences. The rules stay in the services. Everything after a finished order (email, analytics) subscribes to OrderApproved.

Common mistake

A compensation is a rollback

A rollback makes it as if the transaction never happened. Nobody saw it. A compensation is a new transaction, after others may have acted on the first. The customer got a “payment received” email. A report counted the revenue. So compensations are business actions with their own rules (refund fees, an apology email). Some steps cannot be compensated, and must come last. Designing a saga means designing those, not adding an undo() to every method.

Mid-level covers

Knows a transaction cannot span services, names sagas with compensating actions and the outbox pattern, and adds idempotency keys to payment calls.

Senior adds

First asks which data must be atomic, and keeps it in one database. Explains why 2PC blocks, and where it still works (replicated participants). Designs the saga: durable state, keys per step, compensations that retry forever and tolerate arriving first, a pivot, semantic locks, timeouts that never mean “failed”, and reconciliation.

Seen in

• Payment system: state, ledger lines and outbox row in one local transaction. The provider call stays outside it, keyed by attempt id. Voids and refunds undo it, and daily reconciliation checks it.
• Ticket booking: seats HELD as a semantic lock, and the card authorized under the order id. Then a sale plus outbox row, or a void if the hold was lost.
• Hotel reservation: a HELD reservation with a 15 minute expiry, and an explicit path for a payment that outlasts its hold.
• Ride hailing: fare and outbox row commit together. The trip id is the payment’s idempotency key.
• Job scheduler: an outbox read by CDC, and run ids as idempotency keys because execution is at least once.
• Message queue: Kafka’s transactions, two-phase commit inside the log with commit markers in every partition touched.
• Ad click aggregator: compares a two-phase commit sink to Kafka with idempotent upserts of absolute totals, and picks the upserts.

Recap

The problem. Any failure between two writes to two systems leaves them disagreeing. A remote call can time out with no answer. First, put data that must change atomically in one database.

2PC. Prepare, a forced commit record (the commit point), then commit. Atomic and isolated, but participants that voted yes block, holding locks, if the coordinator dies before telling them. Good inside one database whose participants are consensus groups (Spanner, CockroachDB), wrong across services.

Sagas. Local transactions in sequence, compensations in reverse on failure. Atomic in the end, never isolated. Needs durable orchestrator state, idempotent steps, compensations that retry until they land and tolerate arriving first, irreversible steps last, and semantic locks.

TCC. Try reserves with an expiry, Confirm uses, Cancel releases. Card authorize, capture and void. Handle empty cancels, hanging tries and expired reservations.

Every hop. Outbox out, idempotent consumer in: exactly-once effect, at-least-once delivery.

Numbers. 2PC: two round trips with locks held, a few ms in one data center, 140 ms or more across regions. Saga: about 100 to 300 ms for three orchestrated steps over CDC and Kafka.

Further reading

• On this site: message queues for the outbox relay, databases for distributed SQL and CDC, and consistency models for isolation levels.
• Designing Data-Intensive Applications, Martin Kleppmann: atomic commit, 2PC and why it blocks, and the case for end-to-end idempotency instead of distributed transactions.
• Pattern: Saga, microservices.io: orchestration, choreography and the isolation countermeasures, from Chris Richardson’s Microservices Patterns.
• Outbox event router, Debezium documentation: publishing outbox rows from the write-ahead log with CDC.