Design a payment system

“Design the payment system for our marketplace. Buyers pay by card, and we pay the sellers.”

It sounds like plumbing around one call to Stripe. It’s the problem where one careless retry takes real money from a real person. A payment system moves money it doesn’t hold, through companies it doesn’t control. A payment service provider (PSP), such as Stripe or Adyen, charges the card. The money reaches our bank days later, and sellers are paid from there. Our system must know where every cent is and why, even though most of those steps happen across a network call that can time out.

Throughput isn’t the problem: the peak is about 600 payments a second. Correctness is. So this page starts with one service that charges through a PSP, and lets each kind of failure grow it. Then it tests the result: one charge, a busy ledger account, reconciliation, and a second PSP.

The framework

1. 01Answer
2. 02Scope
3. 03Numbers
4. 04Build
5. 05Stress
6. 06Failures
7. 07Wrap up

The answer in one minute

If you only have a minute, this is the design. The rest of the page explains each part.

1. Never touch card numbers. The PSP’s own form takes the card and gives us a token. That keeps us out of most card security rules.
2. Use two idempotency keys. One from the caller, so a retry finds the first payment. One of ours toward the PSP, so our retry can’t charge twice.
3. Write “pending” before every call out, and treat a timeout as unknown. Never mark a timed-out payment failed, because the card may have been charged. A resolver asks the PSP what happened.
4. Keep a double-entry ledger, in the same transaction. Every movement is balanced debits and credits. The payment’s state, its ledger lines and the event others act on commit together.
5. Check everything against the PSP daily. Webhooks are hints, the state machine decides, and a daily reconciliation catches whatever the rest missed.

Scope it

A few questions decide the shape of the design: who touches the card, whether we pay sellers, and when money is taken.

Interviewerminute 0

Design the payment system for our e-commerce marketplace.

What they are testing

Whether you establish that a PSP charges the cards, that sellers are paid out, and when money is captured. About three minutes. Asking only about requests per second reads badly: volume is the least interesting number in this problem.

These four questions matter most:

• Do we process cards, or a PSP? PSPs. We never see card numbers.
• Marketplace, or our own goods? A marketplace: we keep a commission, and sellers get the rest.
• When is money taken? We authorize at checkout (reserve the money on the card) and capture when the order ships (actually take it).
• How many payments? 5 million a day, ten times that on Black Friday.

From these answers we make five promises:

• Charged once. Never two charges for one checkout attempt, and never a charge we didn’t record. The target is zero: at 5 million payments a day, a 0.01% double-charge rate is 500 angry buyers a day.
• The ledger balances. Every entry’s debits equal its credits, and nothing in it is ever edited or deleted.
• Nothing lost. No committed payment is lost when a machine or zone fails. Records are kept 7 years.
• Always up. Taking payments works 99.99% of the time, 52 minutes of downtime a year. One PSP alone can’t give this.
• Fast enough. Authorization under 1.5 seconds at p99, of which our own share is under 100 ms.

We leave out fraud models (the PSP scores risk), tax, currency conversion and subscriptions.

Go deeperMore questions, and what each changes1 min

Question	We assume	If the answer were different
Our own goods only?	A marketplace	Removes seller balances and payouts, half the ledger
Digital goods?	Capture when the order ships	Capture at once: one state fewer, no expiring authorizations
Refunds, disputes, currencies?	Partial refunds, chargebacks, several currencies, no conversion	Converting currencies adds rates, rounding and FX accounts
How fast are sellers paid?	Daily, after a holding period	Instant payouts need a live balance check per payout
100 times the volume?	No	Sharding becomes a day-one decision

The numbers that decide it

Now we work out the numbers. The surprise is how small they are.

Assumptions

Payments a day

5 million

Black Friday peak over average

10×

Write transactions per payment

6

PSP calls per payment

2

Bytes per payment, with ledger

1.5 KB

Years kept for audit

7 years

Sellers paid each day

1 million

Payments60/s600/s at peak

5 million a day ÷ 86,400 s. About 600 a second at the Black Friday peak.

So we a handful of servers, so we can spend everything on correctness: strict transactions and synchronous copies on every payment.

Database writes3,600/sat peak

About 6 small write transactions per payment over its life: create, authorize, capture and webhooks. One Postgres primary on fast disks sustains about 10,000.

So we one Postgres primary with a synchronous standby. Splitting is a ten-times problem.

PSP calls1,200/sat peak

An authorization and a capture per payment. PSPs limit each account, and raise limits on request.

So we agree the peak with the PSP months ahead. It's a contract line, not code.

Storage2.7 TBa year

About 1.5 KB a payment with its attempts, five ledger lines and indexes. 19 TB over 7 years.

So we split tables by month and move closed months to cheap storage. The ledger is append-only, so a closed month never changes.

Money$300Ma day

At $60 an order.

So we PSP fees, 2 to 3% of that, are the cost that matters, far above any server bill.

Try the calculator: double the volume and hot storage forces a split before the write rate does. At ten times, the write rate and the PSP contract give out too.

Change an assumptionReset to the page's numbers

Payments a day

5 million

Black Friday peak over average

10×

Write transactions per payment

6

PSP calls per payment

2

Bytes per payment, with ledger

1.5 KB

Years kept for audit

7 years

Sellers paid each day

1 million

Payments, average

58/s

Payments, peak

580/s

Low for a web service: a few servers carry it. The hard part is correctness, not throughput.

Database write transactions, peak

3,500/s

One Postgres primary takes about 10,000 small write transactions a second: room to spare.

PSP calls, peak

1,200/s

Well inside the 8,000 calls a second we assume one PSP contract allows. Agree the peak with the PSP before Black Friday anyway.

Hot data, 13 months in Postgres

3 TB

Fits one well-provisioned primary. Closed months move to object storage and the warehouse.

Kept for audit

19 TB

Lives in object storage and the warehouse, not in the transactional database.

Payouts in a 4 hour batch window

69/s

A batch, not a hot path. The limit is the bank file format and cut-off times, not our servers.

Build the design

We start with the simplest design that works. Then we let each failure break it, one at a time. The load is small and the risk is large. So every step answers one question: can this cause a second charge, or a charge with no record? Throughout, follow one scenario: a buyer presses Pay.

Step 1. Charge through a PSP

The checkout page loads the PSP’s payment fields, a frame served by the PSP. So the card number goes from the browser straight to the PSP, which returns a token. The order service sends that token to our payment service, which asks the PSP to authorize the amount. When the order ships, it captures.

Payment system · architecture

Stage 1: Charge through a PSP

Color keyServiceDatabaseQueue or streamBackground workerNew in this stageAsync or occasional

Payments per second:

600

60 average600 peak6k (10×)20k

Stage 1 of 7

Charge through a PSP

Why this stage

Nothing yet.

This is the smallest design that takes money for an order.

What we add

The browser sends the card straight to the PSP and gets a token.

One payment service charges that token and records the result in Postgres.

What it costs

A retry can charge twice, a timeout leaves a payment nobody can explain, there is no ledger, and one server is one failure.

At 600 payments per second, Payment service is 150% busy. Stage 2 is the fix.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What one machine can do

• One payment service instance handles about 400 payments a second. Each one holds a PSP call open for about half a second.
• One Postgres primary on NVMe takes about 10,000 small write transactions a second.

What one payment costs

• A payment costs 3 write transactions in stage 1. Once it has a ledger, an outbox and webhooks, it costs 6.
• Each payment makes 2 PSP calls: authorize, then capture.
• Each payment sends 2 webhook events.

Outside limits

• We assume our PSP contract allows about 8,000 calls a second.

It breaks on the first bad day. The order service times out and retries, so the same request arrives twice. A server dies right after calling the PSP, and nobody knows whether the card was charged. And there is no record of where the money went.

Go deeperWhy card numbers stay out1 min

Storing card numbers, even encrypted, would put every system that can reach them under the card industry’s security rules (PCI DSS). That’s hundreds of controls, and a yearly audit. With fields served entirely by the PSP, we qualify for the shortest self-assessment. We keep the token, the card brand, the last four digits and the expiry. Nothing else.

Step 2. Never charge twice

Duplicates come from three places, each with its own defense:

• The caller retries. Its idempotency key is inserted in the same transaction as the payment, with a unique constraint. A retry finds the first payment, even when two retries race on two servers.
• We retry the PSP. Every call out gets an attempt ID, sent as the PSP’s own idempotency key and as its reference. Repeating a call with the same attempt ID can’t charge twice.
• We die mid-call. The attempt is saved as AUTH_PENDING before the call. A crash leaves evidence that a charge might exist.

What to do when the PSP doesn’t answer is the decision that matters most. Marking the payment failed is a double charge: the card was often charged anyway, and the buyer pays again. So we answer 202, “processing”, and a resolver worker asks the PSP about attempts pending for over a minute.

Every *_PENDING state means “we asked and don’t know yet”, written before asking:

The payment state machine

Scroll or pinch to zoom, drag to pan

• Color key
• State
• Fails or stops
• Ends well

Every arrow is a compare-and-set: “set authorized, only if still auth pending”. So when the API response, a webhook and the resolver all report one outcome, only the first changes anything.

JPaymentService.javaPypayment_service.pyC#PaymentService.csGopayment_service.goTSpayment-service.ts

JavaPythonC#GoTypeScript

JavaPythonC#GoTypeScript

// Two keys protect the buyer. The caller's Idempotency-Key means one payment per checkout
// attempt; our attempt id, sent to the PSP as its idempotency key, means one charge per
// attempt however many times the call is retried.
record AuthorizeRequest(String orderId, long amountMinor, String currency, String cardToken) {}
record PaymentView(String paymentId, String status) {}
record Existing(String requestHash, PaymentView view) {}
record PspAnswer(boolean approved, String pspRef) {}

interface PaymentStore {
    // One transaction: the idempotency key, the payment, and an attempt in AUTH_PENDING.
    // Returns the earlier record when the key exists, so a retry never reaches the PSP.
    Optional<Existing> begin(String clientKey, String requestHash, String paymentId,
                             String attemptId, AuthorizeRequest req);
    // UPDATE ... SET status = ? WHERE id = ? AND status = 'AUTH_PENDING'. If a webhook or
    // the resolver recorded the outcome first, this changes nothing.
    void finish(String paymentId, String attemptId, String to, String pspRef);
}

interface Psp {
    // Throws HttpTimeoutException when no answer arrives in time: the outcome is then unknown.
    PspAnswer authorize(String idempotencyKey, AuthorizeRequest req) throws HttpTimeoutException;
}

class PaymentService {
    private final PaymentStore store;
    private final Psp psp;

    PaymentService(PaymentStore store, Psp psp) {
        this.store = store;
        this.psp = psp;
    }

    PaymentView authorize(String clientKey, AuthorizeRequest req) throws Exception {
        String body = req.orderId() + "|" + req.amountMinor() + "|" + req.currency() + "|" + req.cardToken();
        String hash = HexFormat.of().formatHex(
            MessageDigest.getInstance("SHA-256").digest(body.getBytes(StandardCharsets.UTF_8)));
        String paymentId = UUID.randomUUID().toString(), attemptId = UUID.randomUUID().toString();

        // Persist the intent before calling out. If this process dies after the PSP call,
        // the AUTH_PENDING row is how the resolver finds the attempt and asks what happened.
        Optional<Existing> earlier = store.begin(clientKey, hash, paymentId, attemptId, req);
        if (earlier.isPresent()) {
            // Same key, different body is a client bug, not a retry: refuse it (HTTP 422).
            if (!earlier.get().requestHash().equals(hash))
                throw new IllegalStateException("idempotency key reused for a different request");
            return earlier.get().view(); // the first answer, or PROCESSING while it is still unknown
        }

        try {
            PspAnswer answer = psp.authorize(attemptId, req);
            String status = answer.approved() ? "AUTHORIZED" : "DECLINED";
            store.finish(paymentId, attemptId, status, answer.pspRef());
            return new PaymentView(paymentId, status);
        } catch (HttpTimeoutException e) {
            // The card may or may not be charged. Leave the attempt pending for the webhook
            // or the resolver. Retrying here with a new attempt id is how double charges happen.
            return new PaymentView(paymentId, "PROCESSING");
        }
    }
}

Payment system · architecture

Stage 2: Never charge twice

Color keyServiceDatabaseQueue or streamBackground workerNew in this stageAsync or occasional

Play

PreviousNext

Stage

12

Payments per second:

600

60 average600 peak6k (10×)20k

Stage 2 of 7

Never charge twice

Why this stage

Clients retry and networks time out, so the same request arrives twice and some PSP calls never answer.

At the 600 a second peak one server is also 150% busy.

What we add

Idempotency keys from the caller and to the PSP, a state machine persisted before every call out, a stateless fleet, and a resolver that settles payments stuck in a pending state.

What it costs

Callers must handle a 202 "processing" answer, and every payment costs one more write.

At 600 payments per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What one machine can do

• One payment service instance handles about 400 payments a second. Each one holds a PSP call open for about half a second.
• One Postgres primary on NVMe takes about 10,000 small write transactions a second.

What one payment costs

• A payment costs 3 write transactions in stage 1. Once it has a ledger, an outbox and webhooks, it costs 6.
• Each payment makes 2 PSP calls: authorize, then capture.
• Each payment sends 2 webhook events.

Outside limits

• We assume our PSP contract allows about 8,000 calls a second.

Step 3. A ledger that always balances

A balance column answers “how much”, never “why”. In double-entry bookkeeping, every movement is an entry of two or more lines. Each line debits or credits one account, and debits equal credits. Money only moves between accounts, so all balances sum to zero, and a bug that breaks this is caught by arithmetic.

Take capturing a $64.99 order from seller 42, with our 10% commission. It debits “PSP owes us” by 64.99. It credits “we owe seller 42” by 58.49, and “our commission” by 6.50. Nothing is ever edited. A mistake is fixed by a new, reversing entry.

The order service must also hear about the capture. Committing, then publishing to Kafka, is a dual write. If we die between the two, the ledger says captured, but the order never ships. So we use an outbox. The event is written as a row, in the same transaction as the state and the ledger lines. A relay publishes it afterwards.

The outbox: a crash makes an event late or repeated, never lost

Scroll or pinch to zoom, drag to pan

• Color key
• Service
• Queue or stream
• Database

Publishing is at least once, so every consumer ignores an event ID it has already seen.

JCaptureRecorder.javaPyline.pyC#CaptureRecorder.csGoline.goTSdatabase.ts

JavaPythonC#GoTypeScript

JavaPythonC#GoTypeScript

enum Side { DEBIT, CREDIT }

// One line of a journal entry. Amounts are positive integers in minor units; the side says
// which way the money moves. Floats cannot represent 0.10 exactly, so they never touch money.
record Line(String account, Side side, long amountMinor, String currency) {}

final class Journal {
    // An entry is valid only if, for every currency, debits equal credits. Money can then
    // only move between accounts, never appear or vanish. The database checks it again.
    static void checkBalanced(List<Line> lines) {
        if (lines.size() < 2) throw new IllegalArgumentException("an entry moves money between accounts");
        Map<String, Long> totals = new HashMap<>();
        for (Line l : lines) {
            if (l.amountMinor() <= 0) throw new IllegalArgumentException("amounts are positive");
            long signed = l.side() == Side.DEBIT ? l.amountMinor() : -l.amountMinor();
            totals.merge(l.currency(), signed, Math::addExact);
        }
        totals.forEach((currency, total) -> {
            if (total != 0) throw new IllegalStateException("unbalanced " + currency + ": " + total);
        });
    }
}

interface Transaction extends AutoCloseable {
    // UPDATE payments SET status = ? WHERE id = ? AND status = ?. False if another path
    // (a webhook, the resolver) already moved it.
    boolean transition(String paymentId, String from, String to);
    void insertEntry(String entryId, List<Line> lines); // entry id is unique
    void insertOutbox(String paymentId, String eventType, String payload);
    void commit(); // closing without commit rolls everything back
    @Override void close();
}

interface Database { Transaction begin(); }

class CaptureRecorder {
    private final Database db;

    CaptureRecorder(Database db) { this.db = db; }

    boolean recordCapture(String paymentId, String sellerId, long amount, long commission, String currency) {
        List<Line> lines = List.of(
            new Line("psp_receivable", Side.DEBIT, amount, currency),                    // the PSP owes us this
            new Line("seller_payable:" + sellerId, Side.CREDIT, amount - commission, currency), // we owe the seller
            new Line("commission_revenue", Side.CREDIT, commission, currency));          // what we earned
        Journal.checkBalanced(lines);

        try (Transaction tx = db.begin()) {
            // The API response and a webhook can both report this capture. The compare-and-set
            // lets exactly one of them post the entry; the other returns false and does nothing.
            if (!tx.transition(paymentId, "CAPTURE_PENDING", "CAPTURED")) return false;
            // A deterministic entry id is a second guard: a replay hits the unique key and fails.
            tx.insertEntry("capture:" + paymentId, lines);
            tx.insertOutbox(paymentId, "payment.captured",
                "{\"paymentId\":\"%s\",\"amount\":%d,\"currency\":\"%s\"}".formatted(paymentId, amount, currency));
            // All three or none: no event without its ledger entry, no entry without its event.
            tx.commit();
            return true;
        }
    }
}

Follow an authorization and a capture. Almost all of the 480 ms is the PSP and the card’s bank. Our own hops add about 30.

Payment system · architecture

Stage 3: A ledger that always balances

Grow itTrace a request

Color keyServiceDatabaseQueue or streamBackground workerNew in this stageAsync or occasional

Play

PreviousNext

Stage

123

Payments per second:

600

60 average600 peak6k (10×)20k

Stage 3 of 7

A ledger that always balances

Why this stage

Finance needs every seller balance and every cent explained, and the order service must learn about a capture exactly when it happened.

What we add

A double-entry ledger in the payments database, written in the same transaction as the state change and an outbox row.

A relay publishes the outbox to Kafka.

What it costs

Events arrive at least once, so every consumer deduplicates.

The capture transaction grows to a dozen rows.

At 600 payments per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What one machine can do

• One payment service instance handles about 400 payments a second. Each one holds a PSP call open for about half a second.
• One Postgres primary on NVMe takes about 10,000 small write transactions a second.

What one payment costs

• A payment costs 3 write transactions in stage 1. Once it has a ledger, an outbox and webhooks, it costs 6.
• Each payment makes 2 PSP calls: authorize, then capture.
• Each payment sends 2 webhook events.

Outside limits

• We assume our PSP contract allows about 8,000 calls a second.

Payment system · architecture

Trace a request: Authorize at checkout

Authorize at checkoutCapture when shipped

Color keyServiceDatabaseQueue or streamBackground workerNew in this stageAsync or occasionalHop in motionHops doneResponse

Play

PreviousNext

Authorize at checkout: step 1 of 8

Buyers to Order service

The buyer presses Pay. The card number already went from the browser to the PSP, which returned a token. Only the token reaches us.

20 ms of a 1500 ms budget

1. Buyers → Order service20 ms
2. Order service → Payment service2 ms
3. Payment service → Payments DB5 ms
4. Payment service → PSP A450 ms
5. Payment service → Payments DB5 ms
6. Payments DB → Outbox relayasync
7. Outbox relay → Kafkaasync
8. Response

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

Step 4. Hearing back from the PSP

Some outcomes come later. The buyer’s bank asks for an identity check. A refund completes two days after we ask. A chargeback (the buyer disputes the charge with their bank) arrives a month later. These arrive as webhooks, HTTP calls from the PSP to us. Three properties shape how we take them:

• Anyone can send one. Each carries a signature made with a secret only we and the PSP hold. Check it before reading anything, and reject old timestamps, so a captured request can’t be replayed.
• They arrive at least once. Store each one in an inbox keyed by the PSP’s event ID, then answer 200. A redelivery changes nothing.
• They arrive in any order. A refund event can come before its capture. An event applies only if its transition is allowed now. An early one waits.

JWebhookIntake.javaPywebhook_intake.pyC#WebhookIntake.csGowebhook_intake.goTSwebhook-intake.ts

JavaPythonC#GoTypeScript

JavaPythonC#GoTypeScript

interface Inbox {
    // INSERT INTO webhook_inbox (psp, event_id, payload) ... ON CONFLICT DO NOTHING.
    // The primary key (psp, event_id) makes a redelivered event a no-op.
    void insertIfAbsent(String psp, String eventId, String payload);
}

class WebhookIntake {
    // Wide enough for clock skew and the PSP's own queueing, narrow enough that a captured
    // request cannot be replayed tomorrow.
    private static final Duration TOLERANCE = Duration.ofMinutes(5);
    private static final ObjectMapper JSON = new ObjectMapper();

    private final String psp;
    private final SecretKeySpec key;
    private final Inbox inbox;
    private final Clock clock;

    WebhookIntake(String psp, byte[] secret, Inbox inbox, Clock clock) {
        this.psp = psp;
        this.key = new SecretKeySpec(secret, "HmacSHA256");
        this.inbox = inbox;
        this.clock = clock;
    }

    // Returns the HTTP status. Anything but a fast 2xx makes the PSP retry the delivery.
    int receive(String body, String timestampHeader, String signatureHeader) throws Exception {
        long ts;
        try { ts = Long.parseLong(timestampHeader); } catch (NumberFormatException e) { return 400; }
        Duration age = Duration.between(Instant.ofEpochSecond(ts), clock.instant()).abs();
        if (age.compareTo(TOLERANCE) > 0) return 400;

        // The signature covers the timestamp and the exact bytes received. Parse nothing
        // before it checks out: an unsigned body is just text from the internet.
        Mac mac = Mac.getInstance("HmacSHA256");
        mac.init(key);
        byte[] expected = mac.doFinal((ts + "." + body).getBytes(StandardCharsets.UTF_8));
        byte[] given;
        try { given = HexFormat.of().parseHex(signatureHeader); } catch (IllegalArgumentException e) { return 400; }
        // Constant time, so response timing does not leak how many bytes matched.
        if (!MessageDigest.isEqual(expected, given)) return 401;

        // The event id comes from the signed body, never from a header anyone could set.
        String eventId = JSON.readTree(body).path("id").asText("");
        if (eventId.isEmpty()) return 400;

        // Store, then acknowledge. Applying the event happens later from the inbox, so a slow
        // database never makes the PSP time out and resend, and nothing acknowledged is lost.
        inbox.insertIfAbsent(psp, eventId, body);
        return 200;
    }
}

The last new box is the reconciler, which matches each PSP’s daily report against the ledger, line by line. A stress test covers it.

Payment system · architecture

Stage 4: The PSP side of the story

Grow itTrace a request

Color keyServiceDatabaseQueue or streamBackground workerNew in this stageAsync or occasional

Play

PreviousNext

Stage

1234

Payments per second:

600

60 average600 peak6k (10×)20k

Stage 4 of 7

The PSP side of the story

Why this stage

Some outcomes arrive only by webhook (3-D Secure, refunds, disputes), and the PSP and the bank are records of their own that ours can drift from.

What we add

A webhook intake that verifies, deduplicates and stores each event before acting on it, and a daily reconciler that matches the ledger against the PSP settlement files.

What it costs

Two more components, and a finance queue for the breaks that cannot be fixed automatically.

At 600 payments per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What one machine can do

• One payment service instance handles about 400 payments a second. Each one holds a PSP call open for about half a second.
• One Postgres primary on NVMe takes about 10,000 small write transactions a second.

What one payment costs

• A payment costs 3 write transactions in stage 1. Once it has a ledger, an outbox and webhooks, it costs 6.
• Each payment makes 2 PSP calls: authorize, then capture.
• Each payment sends 2 webhook events.

Outside limits

• We assume our PSP contract allows about 8,000 calls a second.

This is enough for a mid-level answer. Tokens, two keys, pending states with a resolver, a double-entry ledger with an outbox, and a webhook inbox. Step 5 and the stress tests are what make it a senior answer.

Go deeperThe API2 min

POST /v1/payments
Idempotency-Key: chk_7f3a9c          # one per checkout attempt, made when the page loads

{ "orderId": "ord_88123",
  "amount": { "minor": 6499, "currency": "USD" },
  "paymentMethod": "pm_tok_1Nx...",   # a PSP token, never a card number
  "captureMode": "manual",
  "splits": [{ "sellerId": "s_42", "minor": 6499, "commissionMinor": 650 }] }

201 Created           { "paymentId": "pay_0123_01JD...", "status": "AUTHORIZED" }
202 Accepted          { "status": "PROCESSING" }
200 OK                { "status": "REQUIRES_ACTION", "redirectUrl": "https://..." }
402 Payment Required  { "status": "DECLINED", "declineCode": "insufficient_funds" }

POST /v1/payments/{id}/capture   Idempotency-Key: ship_5531
POST /v1/payments/{id}/void      Idempotency-Key: cancel_88123
POST /v1/payments/{id}/refunds   Idempotency-Key: rma_901      { "minor": 2000 }

Amounts are whole numbers in the currency’s smallest unit, so floating-point rounding never appears. A retry with the same key returns the first response, or 202 while the first is still running, and never calls the PSP again. The same key with a different body gets 422. REQUIRES_ACTION means the buyer’s bank wants an identity check first.

Don’t use the order ID as the key. An order can have several payments: the first card is declined and the buyer tries another. With the order ID as the key, the second card could never be charged. The key belongs to one checkout attempt, and is replaced only after a final decline.

Go deeperThe tables1 min

CREATE TABLE idempotency_keys (
  client_id TEXT, key TEXT,
  request_hash  TEXT NOT NULL,           -- same key, different body: reject
  payment_id    TEXT NOT NULL,
  response      JSONB,                   -- the first answer, replayed on retries
  PRIMARY KEY (client_id, key)
);
CREATE TABLE payment_attempts (
  attempt_id    TEXT PRIMARY KEY,        -- our key and our reference at the PSP
  payment_id    TEXT NOT NULL,
  kind          TEXT NOT NULL,           -- AUTHORIZE, CAPTURE, VOID, REFUND
  status        TEXT NOT NULL,           -- PENDING, SUCCEEDED, FAILED
  next_check_at TIMESTAMPTZ              -- when the resolver should ask
);
CREATE TABLE ledger_lines (
  entry_id      TEXT NOT NULL,           -- deterministic: capture:pay_0123_01JD...
  line_no       SMALLINT NOT NULL,
  account_id    TEXT NOT NULL,           -- seller_payable:s_42, psp_receivable, ...
  side          CHAR(1) NOT NULL CHECK (side IN ('D', 'C')),
  amount_minor  BIGINT NOT NULL CHECK (amount_minor > 0),
  currency      CHAR(3) NOT NULL,
  PRIMARY KEY (entry_id, line_no)
);
CREATE TABLE webhook_inbox (psp TEXT, event_id TEXT, payload JSONB,
                            PRIMARY KEY (psp, event_id));   -- a redelivery is a no-op

Three rules sit outside the table definitions. The application may only insert and read ledger rows, so no code can edit history. A check at commit time rejects any transaction whose lines don’t balance per currency. And seller balances are a cache, rebuilt from the lines, never the truth.

Database	Fits because	Pick when
Postgres	Transactions across tables, constraints, triggers	Under about 10,000 write transactions a second
Spanner, CockroachDB	Transactions across shards and regions	Writes must survive a region loss with nothing lost
DynamoDB	Conditional writes, small transactions	You want no database to run on AWS
A ledger database	Double-entry built in, very high posting rates	Posting rates far beyond this page

Step 5. Paying the sellers

The last thing missing is the sellers. They are owed the money we hold for them.

A capture doesn’t put money in our bank. The PSP pays us about two days later, minus fees, and refunds and chargebacks follow for weeks. So payouts run on their own clock. Every night, per currency, a batch reads each seller’s balance and pays out money that has arrived and passed a holding period. It keeps back a rolling reserve (say 5% of the last 30 days), so future refunds and chargebacks have funds to come from.

One transaction inserts a payout row, unique per seller and day, and an entry moving the amount from the seller’s account to “payouts in transit”. The payout goes out in a bank file or the PSP’s payout API, with the payout ID as its reference. A rerun is safe: the unique key means no seller is paid twice for a day. A payout that bounces days later is reversed back into the seller’s balance.

Payment system · architecture

Stage 5: Pay the sellers

Grow itTrace a request

Color keyServiceDatabaseQueue or streamBackground workerNew in this stageAsync or occasional

Play

PreviousNext

Stage

12345

Payments per second:

600

60 average600 peak6k (10×)20k

Stage 5 of 7

Pay the sellers

Why this stage

Sellers are owed the money we hold for them.

Paying them is a different flow: daily, in bulk, through bank rails that answer days later.

What we add

A payout batch that reads each seller balance from the ledger, holds back a reserve, and sends one idempotent payout per seller per day.

What it costs

Payouts can bounce days later, and a chargeback after a payout leaves a seller owing us money.

At 600 payments per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What one machine can do

• One payment service instance handles about 400 payments a second. Each one holds a PSP call open for about half a second.
• One Postgres primary on NVMe takes about 10,000 small write transactions a second.

What one payment costs

• A payment costs 3 write transactions in stage 1. Once it has a ledger, an outbox and webhooks, it costs 6.
• Each payment makes 2 PSP calls: authorize, then capture.
• Each payment sends 2 webhook events.

Outside limits

• We assume our PSP contract allows about 8,000 calls a second.

Stress it

The design is complete. Now we test it where payments are hard. Proving one charge. The ledger account every payment touches. Reconciling with the PSP. And ten times the volume, with a second PSP. Only the last one adds boxes.

Exactly one charge

Interviewerminute 20

Convince me a buyer can’t be charged twice.

What they are testing

Whether you walk every source of duplicates and name what stops each, then show how an unknown outcome resolves. “We use exactly-once delivery” is the red flag: no network offers it.

No single mechanism guarantees one charge. A chain does, and every link must hold:

Where a duplicate comes from	What stops it
The buyer double-clicks Pay	One checkout attempt key, made when the page loads
The order service retries	The caller’s key, unique in the database
Two retries race on two servers	The same unique constraint
We retry the PSP call	Our attempt ID as the PSP’s idempotency key
We crash after the PSP call	The pending row, found by the resolver
API response and webhook report one result	Compare-and-set on the state
A webhook or event arrives twice	The inbox key; consumers skip event IDs they’ve seen

The hardest link is the timeout after the call out:

A timeout, a retry, and the resolver

Scroll or pinch to zoom, drag to pan

• Color key
• Service
• Database

The resolver asks. It never charges. It looks the attempt up by the reference we sent. PSPs only keep idempotency keys for a while (Stripe may drop them after 24 hours). After that, only the lookup is safe. “Not found” is trusted only after a grace period, since a request can sit in the PSP’s queue. Sometimes a success surfaces later anyway, for a payment we marked failed. Then it’s voided or refunded automatically.

Follow an unknown outcome to its answer:

Payment system · architecture

Trace a request: PSP times out

Authorize at checkoutCapture when shippedPSP times out

Color keyServiceDatabaseQueue or streamBackground workerNew in this stageAsync or occasionalHop in motionHops doneResponse

Play

PreviousNext

PSP times out: step 1 of 8

Order service to Payment service

POST /v1/payments, as before.

2 ms of a 5000 ms budget

1. Order service → Payment service2 ms
2. Payment service → Payments DB5 ms
3. Payment service → PSP A3000 ms
4. PSP A → Webhook intakeasync
5. Webhook intake → Payments DBasync
6. Resolver → PSP Aasync
7. Resolver → Payments DBasync
8. Response

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

JResolver.javaPystuck_attempt.pyC#Resolver.csGopsp_lookup.goTSpsp-lookup.ts

JavaPythonC#GoTypeScript

JavaPythonC#GoTypeScript

enum Lookup { SUCCEEDED, FAILED, NOT_FOUND }

record StuckAttempt(String attemptId, String paymentId, Instant sentAt, int checks) {}
record Found(Lookup result, String pspRef) {}

interface AttemptStore {
    // Pending attempts whose next check is due: SELECT ... FOR UPDATE SKIP LOCKED, so several
    // resolver copies can run without ever working on the same attempt.
    List<StuckAttempt> duePending(Instant now, int limit);
    // The same compare-and-set the API and the webhook path use: only PENDING can move.
    void resolve(String attemptId, boolean succeeded, String pspRef);
    void checkAgainAt(String attemptId, Instant when);
    void escalate(String attemptId); // a person looks at it; money is never guessed
}

interface PspLookup {
    // Looks the attempt up by the reference we sent with it.
    Found find(String attemptId) throws IOException; // IOException: the PSP is unreachable
}

class Resolver {
    // A request can sit in a PSP's queue for a while before it shows up in lookups. Only
    // after this long does "not found" mean "never happened".
    private static final Duration NOT_FOUND_GRACE = Duration.ofMinutes(15);
    private static final int MAX_CHECKS = 20;

    private final AttemptStore store;
    private final PspLookup psp;
    private final Clock clock;

    Resolver(AttemptStore store, PspLookup psp, Clock clock) {
        this.store = store;
        this.psp = psp;
        this.clock = clock;
    }

    void runOnce() {
        Instant now = clock.instant();
        for (StuckAttempt a : store.duePending(now, 100)) {
            try {
                // Ask, never charge again. The only safe resend is the same request with the
                // same idempotency key; a new key is a second charge.
                Found f = psp.find(a.attemptId());
                switch (f.result()) {
                    case SUCCEEDED -> store.resolve(a.attemptId(), true, f.pspRef());
                    case FAILED -> store.resolve(a.attemptId(), false, f.pspRef());
                    case NOT_FOUND -> {
                        if (Duration.between(a.sentAt(), now).compareTo(NOT_FOUND_GRACE) > 0)
                            // Never reached the PSP. If it surfaces later anyway (a webhook, the
                            // settlement file), the state machine voids or refunds it automatically.
                            store.resolve(a.attemptId(), false, null);
                        else store.checkAgainAt(a.attemptId(), now.plus(backoff(a.checks())));
                    }
                }
            } catch (IOException e) {
                // PSP unreachable: keep asking with backoff, and hand it to a person in the end.
                if (a.checks() >= MAX_CHECKS) store.escalate(a.attemptId());
                else store.checkAgainAt(a.attemptId(), now.plus(backoff(a.checks())));
            }
        }
    }

    // 30 s, 1 min, 2 min ... capped at 30 min, so an outage does not turn into a query storm.
    static Duration backoff(int checks) {
        return Duration.ofSeconds(Math.min(1800, 30L << Math.min(checks, 10)));
    }
}

Common mistake

"Exactly-once delivery solves this"

A sender that gets no acknowledgement can’t tell a lost message from a lost acknowledgement. So it resends (at least once) or doesn’t (at most once). What systems build is an exactly-once effect: at-least-once delivery into a receiver that ignores repeats, which is the chain above.

Go deeperOther ways to handle a PSP timeout1 min

On a PSP timeout	What happens	Verdict
Mark the payment failed	The card was often charged anyway; the buyer pays again	Double charge
Retry with a fresh request	A second authorization on the same card	Double charge
Retry with the same key	The PSP returns the first result, or runs it once	Safe, while the PSP keeps the key
Answer 202, resolve by asking	A webhook or a lookup settles it, usually within a minute	Safe at any age; the pick

Saving state before calling out costs one more write per call, a 202 the caller must handle, and pending states every reader must understand. In return, no charge can exist without a row that says it might.

The ledger, and the account every payment touches

Follow the ledger for an order that is captured, partly refunded and paid out. Then look at the account every payment touches.

One $64.99 order from seller 42, with a 10% commission and a PSP fee of 2.9% plus 30 cents:

Entry	Debit	Credit
1. Capture	PSP owes us 64.99	Seller 42 owed 58.49, commission 6.50
2. PSP pays us, 2 days later	Cash 62.81, PSP fees 2.18	PSP owes us 64.99
3. Partial refund of 20.00	Seller 42 owed 18.00, commission 2.00	PSP owes us 20.00
4. Payout to seller	Seller 42 owed 40.49	Payouts in transit 40.49
5. Bank confirms	Payouts in transit 40.49	Cash 40.49

Every row balances, and the seller’s account ends at zero. Four rules keep the ledger trustworthy:

• Append only, enforced by permissions. A wrong entry is fixed by a reversing entry plus a correct one, each with a reason. Manual ones need a second person to approve.
• Balanced per currency. A dollar debit never balances a euro credit.
• Whole numbers in the smallest unit, with one written rounding rule. Commission is rounded, and the seller gets the remainder. So every entry balances to the cent.
• Balances are derived, and rebuilt from the lines nightly to prove they haven’t drifted.

The last rule hides a bottleneck that arrives long before the database is busy. If every capture updates the “PSP owes us” balance row, every capture holds that row’s lock until it commits. At 2 to 5 ms a transaction, one row allows only a few hundred updates a second, under our 600 peak.

So house accounts (PSP owes us, commission) keep no live balance. Entries only add lines, and a job sums them every minute. Nobody needs the commission total to the millisecond. Sellers keep a live balance row, because one seller rarely sees more than a few captures a second. The largest sellers get sub-accounts: several copies of the account, each entry picking one, so the lock is shared.

Go deeperWhy not a separate ledger service?1 min

Large companies run one, and with several products moving money, it’s right. Then a state change and its ledger entry are no longer one transaction. Postings go through an API keyed by entry ID, so repeats are harmless, and a job compares the two sides. With one product, that disagreement is avoidable, so the ledger stays beside the payments.

Kafka’s “exactly-once” doesn’t fix the dual write either. Kafka transactions make writes to several Kafka partitions atomic. They can’t include a Postgres commit. Only one transaction can make two writes atomic, which is why the event goes into the database first.

Reconciliation

The ledger says the PSP owes us $2,100,000 for yesterday. The PSP’s report says $2,098,400. What happens next?

We, the PSP and the bank each keep a record, and no transaction spans them. Reconciliation proves daily that the three agree, and catches every bug the rest of the design missed.

Daily reconciliation of one PSP file

1. Load

The day’s report from the PSP lands in storage. Its hash is recorded, so loading it twice does nothing.

2. Normalize

Each PSP’s format becomes one row shape: our reference (the attempt ID), the PSP’s reference, type, gross, fee, net, currency and date.

3. Match

Join on our reference against ledger entries within a few days, because cut-off times and time zones differ. Compare type, amount and currency.

4. Sort and fix

Every mismatch becomes a “break” with a class: an automatic action, or an owner and a deadline. Fixes are new entries, never edits. Last, the PSP’s payout is matched to the bank deposit.

Go deeperKinds of mismatch, and what happens to each1 min

Break	Usual cause	Action
In the PSP’s file, missing or failed in ours	A wrongly resolved unknown; a lost webhook	Post the missing entry, or refund; file a bug
In ours, missing from the PSP’s file	Settled after the cut-off	Carry forward; a break only after 3 days
Fee differs from the agreed fees	A contract change, or a PSP pricing error	Post the actual fee; finance claims any overcharge
PSP payout differs from the bank deposit	A reserve held by the PSP, bank fees	Treasury looks into it

The $1,600 in the question is usually a few such lines. A fee at a different rate. A chargeback taken before its webhook arrived. A capture after the cut-off. Each becomes an entry. Health is two numbers: the share matched automatically (above 99.9%), and the oldest open break (under 3 days).

Ten times the volume, and a second PSP

Volume grows ten times, and the PSP goes down for an hour on Black Friday. What do we wish we had built?

At ten times, the database needs 36,000 write transactions a second, or 3.6 primaries. We split by payment. So a payment and everything it writes (attempts, ledger lines, outbox rows) live on one database, and every guarantee above survives. The idempotency key arrives before any payment ID exists. So the database is chosen by hashing the key, and the payment ID is minted with that choice inside it. Webhooks find the right database from the attempt ID the PSP echoes.

Payment system · architecture

Stage 6: Shard by payment

Grow itTrace a request

Color keyServiceDatabaseQueue or streamBackground workerNew in this stageAsync or occasional

Play

PreviousNext

Stage

123456

Payments per second:

600

60 average600 peak6k (10×)20k

Stage 6 of 7

Shard by payment

Why this stage

At 10 times the volume, 36,000 write transactions a second is 3.6 times one primary, and the hot data outgrows one node at only twice the volume.

What we add

16 shards.

The shard is chosen from the idempotency key, so a payment, its attempts, its ledger lines and its outbox rows always commit on one shard.

House accounts become one sub-account per shard.

What it costs

A seller balance is now a sum over 16 shards, and a migration we have to run with live money.

At 600 payments per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What one machine can do

• One payment service instance handles about 400 payments a second. Each one holds a PSP call open for about half a second.
• One Postgres primary on NVMe takes about 10,000 small write transactions a second.

What one payment costs

• A payment costs 3 write transactions in stage 1. Once it has a ledger, an outbox and webhooks, it costs 6.
• Each payment makes 2 PSP calls: authorize, then capture.
• Each payment sends 2 webhook events.

Outside limits

• We assume our PSP contract allows about 8,000 calls a second.

“PSP owes us” is touched by every payment, so every account becomes one sub-account per database. Each entry posts within its database, and a balance is the sum of 16 sub-accounts. A nightly job checks that everything still sums to zero, per currency.

A second PSP. One PSP is one failure domain. An hour-long outage on Black Friday is an hour without sales, more than the 52 minutes a year we allow. So a router picks a PSP for each new payment, by card country, currency and cost. It moves new payments away from an unhealthy one. Three rules keep failover from causing the double charge it should prevent:

• An unknown outcome never fails over. If PSP A timed out, A may have charged the card. That payment waits for A’s webhook or the resolver, however long A is down.
• Only definite failures move: a refused connection, or an explicit “not processed”. A decline is the card’s answer, not the PSP’s failure.
• Tokens belong to one PSP. Cards must work at both. That means card-network tokens, tokenizing with both at checkout, or a certified vault that forwards to either.

Payment system · architecture

Stage 7: Two PSPs

Grow itTrace a requestBreak it

Color keyServiceDatabaseQueue or streamBackground workerNew in this stageAsync or occasional

Play

PreviousNext

Stage

1234567

Payments per second:

600

60 average600 peak6k (10×)20k

Stage 7 of 7

Two PSPs

Why this stage

One PSP is one failure domain: an hour of PSP outage on Black Friday is an hour of lost sales.

At 10 times, 12,000 PSP calls a second is also past what we assume one contract allows.

What we add

A second PSP and routing inside the payment service: by cost and approval rate normally, away from a PSP whose error rate spikes.

What it costs

Cards must be tokenized for both PSPs, two reconciliations, and a strict rule that a payment with an unknown outcome never fails over.

At 600 payments per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What one machine can do

• One payment service instance handles about 400 payments a second. Each one holds a PSP call open for about half a second.
• One Postgres primary on NVMe takes about 10,000 small write transactions a second.

What one payment costs

• A payment costs 3 write transactions in stage 1. Once it has a ledger, an outbox and webhooks, it costs 6.
• Each payment makes 2 PSP calls: authorize, then capture.
• Each payment sends 2 webhook events.

Outside limits

• We assume our PSP contract allows about 8,000 calls a second.

Go deeperOther split keys, and other routing1 min

Option	Wins	Loses
Split Postgres by payment	Every transaction stays on one database	Balances span databases; a migration
Spanner or CockroachDB	Transactions across databases	Slower commits, a new system to run, a migration anyway
Split by seller	A seller’s balance in one place	A cart with two sellers splits one entry across machines

Routing	Wins	Loses
Second PSP only when the first is down	Simple	The cold path fails when first needed
Both active, by rules, with failover	Both paths warm; lower fees	Two integrations and reconciliations
Retry declines at the other PSP	Recovers a few soft declines	Card networks penalize repeated retries

Routing on cost pays: 0.1% of $300 million a day is $300,000 a day. The split is triggered when writes pass 60% of a primary at peak, or hot data passes about 5 TB. The move compares every account’s ledger totals to the cent, before reads switch over.

When things break

Interviewerminute 41

The PSP stops answering in the middle of a payment. What happens to the money?

What they are testing

Whether you treat a timeout as unknown, keep the payment pending, and retry only with the same key.

The rule for every failure: no money moves unless we can record it, and an unknown outcome stays unknown until someone who knows tells us.

The PSP stops answering. The attempt was saved as AUTH_PENDING before the call, so the record says “we asked, and don’t know yet”. The buyer sees “processing”, not “failed”, because the card may well have been charged. The resolver asks the PSP again every minute, always with the same attempt ID, so asking twice can never charge twice. When the PSP comes back, its answer or its webhook moves the payment on. This payment never moves to the second PSP, because the first may have charged the card. New payments do move there, once the router sees the error rate climb.

The database fails mid-capture. Each primary copies every commit to a standby in another zone, before answering. So a machine or zone loss loses nothing committed. Checkout fails for about 30 seconds, while the standby takes over. A capture that was in flight is either committed on the standby, or sits in a pending attempt that the resolver closes. There’s no third place for a charge to hide.

Click any box to break it and see what buyers and sellers would notice:

Payment system · architecture

Break it: what fails, who notices, what users see

Color keyServiceDatabaseQueue or streamBackground workerNew in this stageAsync or occasional

Break

A payment service instancePayments DB primaryPSP APSP BResolverOutbox relayKafkaWebhook intakeReconcilerSettlement filesPayout batchBank railsOrder service

Failure: click any box to break it

Payments DB primary goes down

What breaks

No payment can change state, and no ledger entry can be written.

How we notice

Write errors; failover alarm.

What users see

Checkout fails for about 30 seconds.

We fail closed: no charge is made that we cannot record.

What we do

Promote a synchronous standby in another zone, so no committed payment is lost.

With two standbys and quorum commit, losing one never stops writes.

Callers retry with their keys.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

Common mistake

"Use two-phase commit between our database and the PSP"

Two-phase commit needs every participant to prepare and commit on request, and an HTTP API can’t. This design is a saga instead: local transactions, idempotent calls out, undo actions (a void, a refund) for failures, and reconciliation to prove the outcome.

Go deeperEvery failure, and what users see1 min

A whole region. Across regions, copies are asynchronous. So a region loss can lose the last seconds of commits, including attempts already sent. The PSP holds a second record. Before the surviving region takes payments, a job pulls the PSP’s transactions for that window. They carry our attempt ID and the caller’s key, so the job can restore the missing rows. Then a retried key finds its payment, instead of charging again.

Part	Failure	What users see	What we do
Payment service	Dies mid-call	A few errors or 202s	Callers retry with keys; the resolver settles pending attempts
Database primary	Down	Checkout fails for about 30 s	Promote the synchronous standby: nothing lost
A PSP	Outage or slow	New payments move to the other PSP in a minute	A circuit breaker; pending attempts wait for their own PSP
Outbox relay, Kafka	Down	Orders marked paid late; charging fine	The outbox holds every event until it drains
Webhook intake	Down	Later outcomes arrive late	PSPs redeliver for days
Reconciler, payouts	Late or failed	Books close late; sellers paid a day late	Rerun: every step is safe to repeat

Under stress, protect authorizations first, then captures (an authorization lasts days), then refunds, then payouts. Without the database, we fail closed. Taking orders now and charging later is a business decision, not a default.

Operation	Guarantee	What users can see
Create a payment under a key	Exact: unique constraint on the primary	A retry always returns the first payment
Change a payment’s state	Exact per payment: compare-and-set	Racing reports of one outcome change it once
Post a ledger entry	With the state change, balanced, append-only	Every balance explainable line by line
Events to other services	At least once, in order per payment, under a second	An order shows “paid” a moment after capture
Our records against the PSP’s	Reconciled daily	Rarely, a status is corrected the next day

Go deeperRunning it: alerts, security and abuse1 min

Wake someone up when:

• The approval rate per PSP, card brand and country drops 5 points against last week. (Broken integrations often show as declines, not errors.)
• Our own API’s success drops below 99.95%.
• More than 100 attempts sit pending for over 15 minutes.
• Any entry doesn’t balance. Payouts stop until it’s explained.
• The outbox, inbox or reconciliation falls behind.

Security. PSP keys and webhook secrets live in a key management service and rotate. Manual ledger fixes need a second person and an audit entry. Refunds go only to the original card, never above the captured amount.

Card testing. Bots run stolen cards through checkout with small amounts to find live ones. Limit attempts per account, device and address, challenge after repeated declines, and use the PSP’s risk score.

Cost is PSP fees, not servers: each point of approval rate is $3 million a day of sales.

Wrap up

The whole design

Here is everything we built, in the order we built it. Use the buttons to step through it.

Payment system · architecture

The full design, stage by stage

Color keyServiceDatabaseQueue or streamBackground workerNew in this stageAsync or occasional

Play

PreviousNext

Stage

1234567

Payments per second:

600

60 average600 peak6k (10×)20k

Stage 7 of 7

Two PSPs

Why this stage

One PSP is one failure domain: an hour of PSP outage on Black Friday is an hour of lost sales.

At 10 times, 12,000 PSP calls a second is also past what we assume one contract allows.

What we add

A second PSP and routing inside the payment service: by cost and approval rate normally, away from a PSP whose error rate spikes.

What it costs

Cards must be tokenized for both PSPs, two reconciliations, and a strict rule that a payment with an unknown outcome never fails over.

At 600 payments per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.The whole design. Step through it: charging through a PSP, never charging twice, a ledger that balances, the PSP side of the story, seller payouts, a split by payment, then a second PSP.

The numbers behind the busy bars

What one machine can do

• One payment service instance handles about 400 payments a second. Each one holds a PSP call open for about half a second.
• One Postgres primary on NVMe takes about 10,000 small write transactions a second.

What one payment costs

• A payment costs 3 write transactions in stage 1. Once it has a ledger, an outbox and webhooks, it costs 6.
• Each payment makes 2 PSP calls: authorize, then capture.
• Each payment sends 2 webhook events.

Outside limits

• We assume our PSP contract allows about 8,000 calls a second.

The decisions, and what each one costs

Decision	We chose	We gave up	We’d choose differently if
Card data	PSP-hosted fields and tokens	Control over raw card data	We had to route raw cards ourselves
Unknown outcomes	Answer 202, resolve by asking the PSP	A final answer on every request	The PSP guaranteed a synchronous answer
Ledger	Double-entry, append-only, beside the payments	An independent ledger service	Several products moved money
Scaling	Postgres split by payment, sub-accounts per database	Single-row balances	Writes had to survive a region loss (Spanner)
Payouts	A daily batch with a reserve	Instant payouts	Sellers paid for speed
PSPs	Two, both active, failover only on definite failures	One integration	Volume were small enough for one PSP

How to spend 45 minutes

• 0 to 4. PSP, marketplace, capture timing. Correctness as “zero double charges”.
• 4 to 8. 600 payments a second at peak, and what being small buys.
• 8 to 20. Build it: tokens, two keys, pending states and the resolver, the ledger and outbox, webhooks, payouts. A mid-level answer can stop before payouts.
• 20 to 30. The stress tests: exactly one charge, then the ledger.
• 30 to 41. Reconciliation, then the split and a second PSP.
• 41 to 45. The PSP going silent mid-payment, then your own questions.

Never skip the timeout, the state machine, or the outbox.

Follow-up questions

Interviewer asksA buyer says they were charged twice. How do you investigate?

List the buyer’s payments for that order. Two payments under two checkout keys are two attempts. That’s often a retry after a decline, and a product bug if Pay was offered again while the first was processing. Two charges under one key would be a broken link in our chain. Often it’s neither. An authorization not yet released looks like a second charge on a bank statement.

Interviewer asksHow does a chargeback hit the ledger and the seller?

The PSP takes the money when the dispute opens, often before its webhook arrives. The entry debits the seller’s account and credits “PSP owes us”. A negative seller balance is taken from future payouts, which is what the reserve is for. Winning the dispute posts a reversing entry.

Go deeperMore follow-up questions1 min

How would you move to another PSP without buyers re-entering cards? Most PSPs will export stored cards directly to another certified PSP on request. So card numbers never pass through us. Map old tokens to new, and move new payments over by country. Keep the old PSP live for captures, refunds and disputes of its payments, for months.

How do you test this without charging real cards? PSP sandboxes have test cards for approvals, declines, identity checks and disputes. A fake PSP of our own injects timeouts, and repeated or reordered webhooks. The resolver is tested from a crash between every pair of steps.

Product wants a wallet: buyers keep a balance with us. Technically it’s a ledger account per buyer that payments debit. The real change is legal: holding customer money usually needs an e-money or money transmitter licence.

Further reading

• Distributed transactions and sagas, message queues and consistency models on this site, and API design for idempotency keys in general.
• Designing robust and predictable APIs with idempotency, Stripe engineering: idempotency keys on both sides of a payment API.
• Pattern: Transactional outbox, microservices.io: the outbox and its relay options.