Message queues and streams

The problem it solves

A shop takes 2,000 orders a second at the evening peak. After the order commits, checkout calls five other systems before it answers: the email provider, inventory, fraud scoring, loyalty points and analytics. Each one is a problem.

• Latency adds up. The email provider alone has a p99 of 1.5 seconds. The shopper waits for the slowest of five calls, and for an email they will read later.
• Availability multiplies. Five dependencies at 99.9% each give checkout 0.999⁵ = 99.5%. That’s about 3.6 hours of failed checkouts a month, caused by systems that have nothing to do with taking money.
• Bursts pass straight through. A flash sale brings 20,000 orders a second for 10 minutes. The email provider accepts 5,000 a second, so 15,000 calls a second fail, and with them the orders.
• Retries have nowhere to live. Say the email provider is down for 20 minutes. The request handler can retry for a few seconds at most. Then the email is gone.

A message queue puts a durable buffer between the code that knows something happened (the producer) and the code that acts on it (the consumer). Checkout commits the order, hands the broker a message, and answers in 30 ms. Consumers take messages at their own pace. That buys three things:

What it buys	What happens	In the example
Decoupling	The producer does not know or wait for who consumes	Checkout no longer fails when the email provider fails
Buffering	Bursts become a backlog that drains at the consumers’ rate	9 million emails queue up during the sale and drain in about 50 minutes
Retries	A failed message stays in the broker and is tried again later	The 20 minute outage delays emails; it loses none

The price: the work happens later and somewhere else, failures surface minutes later in another service, and every message can arrive twice. Anything the response depends on, such as “was the card authorized”, stays synchronous.

Checkout before and after a queue

Scroll or pinch to zoom, drag to pan

• Color key
• Service
• Database
• Cache
• Queue or stream
• Background worker

How it works

Step 1: A durable buffer

The simplest queue is a list in the producer’s memory, such as a Go channel. Everything in it dies with the process, and every deploy restarts every process. So the buffer must live outside the producer and survive a crash.

A table in the database works, and is often the right first answer. Insert a row, and workers claim rows with SELECT ... FOR UPDATE SKIP LOCKED. One Postgres instance handles a few thousand jobs a second this way. Past that, or when many teams share it, a dedicated broker takes over. It stores messages on disk, and replicates them across machines. It answers the producer only once the message is safe.

That answer is the first guarantee in the system. Amazon SQS replies after storing the message redundantly across availability zones. Kafka with acks=all replies after every in-sync replica has it. A producer that gets no reply must send again. So even the producer side creates duplicates. Kafka’s idempotent producer (on by default since 3.0) drops those retries, by a per-partition sequence number. SQS FIFO drops a repeated deduplication id within 5 minutes. Neither helps when the application itself sends twice.

Step 2: Acknowledge after the work

If the broker deleted a message as soon as it handed it out, a consumer that crashed one line later would lose it. So a work queue (SQS, RabbitMQ, ActiveMQ, Azure Service Bus) hands a message out and keeps it:

1. Receive. The broker gives the message to one consumer. It hides it from the others for the visibility timeout (SQS: 30 seconds by default, up to 12 hours).
2. Ack. After doing the work, the consumer acks, and only then is the message deleted (DeleteMessage in SQS, basic.ack in RabbitMQ).
3. Redeliver. With no ack, because the consumer crashed or hung, the message becomes visible again for another consumer. SQS waits for the timeout. RabbitMQ redelivers when the consumer’s channel closes.

A consumer crashes before its ack, and the message comes back

Scroll or pinch to zoom, drag to pan

• Color key
• Background worker
• Queue or stream

Nothing was lost, and order 105 was charged twice. Step 6 deals with that.

Two settings decide how well this works:

• The visibility timeout must be longer than the slowest normal processing time, several times the p99. Too short, and a slow but healthy consumer has its message handed to a second consumer while it is still working. For long jobs, the consumer extends the timeout as it goes (a heartbeat: ChangeMessageVisibility in SQS).
• Prefetch is how many unacked messages one consumer may hold (basic.qos in RabbitMQ, up to 10 per receive in SQS). It is the consumer’s backpressure: a slow consumer stops getting more until it acks what it has.

Step 3: Retries, poison messages and dead letters

Redelivery is also the retry mechanism. A consumer that gets a 503 from the email provider does not ack, and the message comes back after the timeout.

Some messages fail every time: a malformed payload, an order id that no longer exists, or a bug that throws on one input. This is a poison message. Retried forever, it eats consumer time. In an ordered queue, it blocks everything behind it.

So the broker counts receives. After a limit (SQS maxReceiveCount, RabbitMQ quorum queue delivery-limit), it moves the message to a dead-letter queue (DLQ). That’s a separate queue nobody consumes automatically. A person can inspect the message there, fix the cause, and redrive it back to the main queue.

Good consumers also classify errors themselves. A validation error goes to the DLQ at once, since a retry cannot fix it. A timeout is retried with exponential backoff:

JWorkQueue.javaPywork_queue.pyC#WorkQueue.csGowork_queue.goTSwork-queue.ts

JavaPythonC#GoTypeScript

JavaPythonC#GoTypeScript

import java.util.ArrayList;
import java.util.List;
import java.util.Optional;

class Message {
    final String id;
    final String body;
    int receives = 0;
    long visibleAt = 0; // while a consumer holds it, hidden until this time (seconds)
    long receipt = 0; // changes on every receive, so only the latest holder can ack

    Message(String id, String body) {
        this.id = id;
        this.body = body;
    }
}

record Delivery(Message message, long receipt) {}

// The core of an SQS or RabbitMQ queue. A received message is hidden, not removed: only an ack
// removes it. If the consumer dies, the message reappears when its visibility timeout runs out,
// so a crash delays work instead of losing it.
class WorkQueue {
    private final long visibilityS;
    private final int maxReceives;
    final List<Message> deadLetters = new ArrayList<>();
    private final List<Message> messages = new ArrayList<>();
    private long receipts = 0;

    WorkQueue(long visibilityS, int maxReceives) {
        this.visibilityS = visibilityS;
        this.maxReceives = maxReceives;
    }

    synchronized void send(String id, String body) {
        messages.add(new Message(id, body));
    }

    synchronized boolean isEmpty() {
        return messages.isEmpty();
    }

    synchronized Optional<Delivery> receive(long now) {
        for (Message m : List.copyOf(messages)) {
            if (m.visibleAt > now) continue; // in flight with another consumer
            if (m.receives >= maxReceives) {
                // It failed every attempt: a poison message. Park it so it stops eating
                // consumer time, and keep it for a person to inspect and redrive.
                messages.remove(m);
                deadLetters.add(m);
                continue;
            }
            m.receives++;
            m.visibleAt = now + visibilityS;
            m.receipt = ++receipts;
            return Optional.of(new Delivery(m, m.receipt));
        }
        return Optional.empty();
    }

    // A heartbeat for slow work: keep the message hidden while still working on it. Without
    // this, a job longer than the timeout is handed to a second consumer.
    synchronized boolean extend(String id, long receipt, long now) {
        for (Message m : messages) {
            if (m.id.equals(id) && m.receipt == receipt) {
                m.visibleAt = now + visibilityS;
                return true;
            }
        }
        return false;
    }

    // The receipt proves this consumer still holds the message. One whose timeout ran out, and
    // whose message went to someone else, gets false and must not assume its work was the only
    // run.
    synchronized boolean ack(String id, long receipt) {
        return messages.removeIf(m -> m.id.equals(id) && m.receipt == receipt);
    }
}

public class Main {
    public static void main(String[] args) {
        WorkQueue q = new WorkQueue(30, 3);
        q.send("m1", "charge order 101");
        q.send("m2", "corrupt payload");
        q.send("m3", "charge order 103");

        long now = 0;
        boolean crashed = false;
        while (!q.isEmpty()) {
            Optional<Delivery> got = q.receive(now);
            if (got.isEmpty()) {
                now++; // everything left is in flight: wait for a timeout to run out
                continue;
            }
            Message m = got.get().message();
            long receipt = got.get().receipt();
            if (m.id.equals("m1") && !crashed) {
                crashed = true; // the consumer dies holding m1, before its ack
                System.out.printf("t=%ds %s received, consumer crashes%n", now, m.id);
                continue;
            }
            if (m.body.equals("corrupt payload")) {
                System.out.printf("t=%ds %s fails, attempt %d%n", now, m.id, m.receives);
                continue; // no ack: it comes back after the timeout
            }
            System.out.printf("t=%ds %s done, acked %b%n", now, m.id, q.ack(m.id, receipt));
        }

        // m3 finished before m1: redelivery after a crash reorders a plain queue.
        System.out.println("dead letters: " + q.deadLetters.stream().map(m -> m.id).toList());
        // t=0s m1 received, consumer crashes
        // t=0s m2 fails, attempt 1
        // t=0s m3 done, acked true
        // t=30s m1 done, acked true
        // t=30s m2 fails, attempt 2
        // t=60s m2 fails, attempt 3
        // dead letters: [m2]
    }
}

The output shows a third property of work queues: m3 finished before m1, because m1 went back into the queue after the crash. Redelivery reorders a plain queue.

Step 4: Keep the messages: a log

A work queue deletes a message once it is processed. That fits jobs: send this email, resize this image. It fits events badly. Say inventory, analytics and fraud all want every order event. A queue needs a copy per consumer. And when analytics fixes a bug and wants to reprocess last week, the messages are gone.

A log (Kafka, Amazon Kinesis, Redpanda, Apache Pulsar) keeps every message for a retention period, whether or not anyone read it. Kafka’s default is 7 days. Messages are appended to a partition, an ordered file. Each gets the next offset: 0, 1, 2 and so on. The broker keeps no per-message state. Instead, each consumer remembers one number per partition. That’s the offset it has processed up to, which it commits back to the broker.

A topic with three partitions, read by two consumer groups

Scroll or pinch to zoom, drag to pan

• Color key
• Service
• Database
• Cache
• Queue or stream
• Background worker

Three properties follow from storing offsets instead of deleting messages:

• Consumer groups. Each group gets every message. Within a group, each partition is read by exactly one member. So a group scales out to at most as many members as there are partitions. The email group and the analytics group above read the same data, at different positions. And a slow group does not slow the other.
• Replay. Resetting a group’s offset to yesterday reprocesses yesterday. A new consumer can start from the beginning of retention and build its state from history.
• Cheap reads, coarse acks. A consumer acknowledges a position, not a message. It cannot skip one bad message and come back to it later. Everything before the committed offset is done.

Design a message queue covers how the broker stores and replicates partitions.

Step 5: Ordering, and what it costs

A work queue with competing consumers has no useful order. Two consumers take m1 and m2 at the same moment, m2 finishes first, and a retried m1 can come back minutes later. If m1 is “order created” and m2 is “order cancelled”, the cancellation is applied to an order that does not exist yet.

Order is kept by giving up parallelism for related messages:

• Kafka and Kinesis keep order within a partition. The producer sends a key. The partition is hash(key) mod partitions. So every event for order 101 lands on the same partition, and is read in order by one consumer.
• SQS FIFO queues keep order per message group id. They hand out a group’s next message only after the previous one is acked, or expires. Different groups proceed in parallel.
• RabbitMQ keeps order in a queue with a single consumer. Per-key order needs a consistent-hash exchange, or one queue per shard of keys.

Order per key (one order, one account, one conversation) is almost always what the product needs. Order across all keys means one partition and one consumer, and that consumer’s throughput.

Per-key order has costs that show up in production:

Cost	What happens
Head-of-line blocking	A slow or failing message stalls every message behind it in its partition or group, including unrelated keys that hashed there
Parallelism cap	A group has at most one active consumer per partition. Extra consumers sit idle
Hot keys	One key with 20% of traffic makes its partition the bottleneck, and adding partitions does not split a key
Repartitioning	Adding partitions changes hash(key) mod N, so a key’s new events land elsewhere while old ones are still being read

Step 6: Delivery semantics, stated precisely

Whether the consumer acks before or after the work decides how the crash in step 2 ends. The guarantees are about effects after failures: how many times the work takes effect when consumers, brokers and networks fail at the worst moment.

Guarantee	How	After a crash at the worst moment	Fits
At most once	Ack (or commit the offset) first, then process	The message is lost	Metrics samples, presence pings, anything lossy
At least once	Process first, then ack	The work happens twice	The default for almost everything
Effectively once	At least once, plus a consumer that recognizes repeats	Delivered twice, takes effect once	Payments, inventory, emails, any side effect

Try both orders. The consumer crashes once while handling m5, in the gap between its two steps. Run Charge, then ack on the queue, then switch to Ack, then charge, then turn on the idempotent consumer. Then do the same on the log, and replay it from offset 0.

Work queueLog

Ack, then chargeCharge, then ack

Idempotent consumer (skip ids already charged) Crash once while handling m5 m7 is poison (fails every time)

RunCrash the consumer nowReset

t = 0 s

Queue

1. m1
2. m2
3. m3
4. m4
5. m5
6. m6
7. m7
8. m8
9. m9
10. m10

Consumer idle

Charges made

1. m10
2. m20
3. m30
4. m40
5. m50
6. m60
7. m70
8. m80
9. m90
10. m100

Charged once

0

Lost

0

Charged twice or more

0

Duplicates skipped

0

Dead-lettered

0

Charge order: none yet

Acking first loses m5, and no amount of deduplication brings it back. Acking after charges m5 twice. On the log, it is worse. The consumer commits once per batch, so a crash replays everything since the last commit, and m4 is charged twice too. Only at-least-once delivery with an idempotent consumer gives one charge per message. And it survives a full replay.

Exactly-once delivery, in the literal sense, means every message handed over once and only once. No system can promise that. The ack itself travels over a network. If it is lost, the broker cannot tell “processed, ack lost” from “never processed”. It must choose between redelivering and dropping. What systems do offer is exactly-once effects, by making a repeated delivery harmless. There are three ways:

• Make the operation idempotent by nature. SET status = 'shipped' can run twice. balance = balance - 10 cannot. Write absolute values, upserts keyed by the event id, or conditional writes (UPDATE ... WHERE version = 7) that fail the second time.
• Record what you processed, in the same transaction as the effect. An inbox table of applied message ids sits next to the consumer’s data, and a repeat hits its unique key. A log consumer can store its offset in that transaction instead, and seek to it on restart. But that misses a repeat the producer wrote at a new offset.
• Use the broker’s transactions, inside the broker only. Kafka transactions commit a stream processor’s output records and its input offsets atomically. That gives exactly once, from one topic to another. Anything outside Kafka, an email or a row in Postgres, is not covered. The message queue problem walks through the mechanism.

The inbox on a log: one transaction per record, and an offset commit only after the whole batch. A crash before the commit replays the batch, and each repeat costs a lookup:

JMain.javaPycrash_error.pyC#Program.csGoloyalty_db.goTScrash-error.ts

JavaPythonC#GoTypeScript

JavaPythonC#GoTypeScript

import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.StringJoiner;
import java.util.TreeMap;

// eventId is set by the producer, so a republished event keeps its id.
record LogRecord(long offset, String eventId, int orderId, int points) {}

// One partition of a log. The broker keeps every record and one number per consumer group: the
// committed offset, where the group resumes after a crash or a rebalance.
class Partition {
    final List<LogRecord> records;
    long committed = 0;

    Partition(List<LogRecord> records) {
        this.records = records;
    }

    List<LogRecord> poll(long start, int maxRecords) {
        int from = (int) Math.min(start, records.size());
        return records.subList(from, Math.min(from + maxRecords, records.size()));
    }
}

// The consumer's own database. The inbox table of applied event ids lives next to the balances
// it protects, so the dedupe record and the effect commit together.
class LoyaltyDb {
    final Map<Integer, Integer> points = new HashMap<>();
    final Set<String> inbox = new HashSet<>(); // PRIMARY KEY (event_id), pruned after a few days

    synchronized boolean apply(LogRecord r) { // BEGIN ... COMMIT
        // Dedupe by event id, not by offset: a relay that republishes after a crash writes the
        // same event at a new offset, and only the id shows it is a repeat.
        if (!inbox.add(r.eventId())) return false; // seen before: the effect already happened
        points.merge(r.orderId(), r.points(), Integer::sum);
        return true;
    }
}

class CrashException extends RuntimeException {
    CrashException(String message) {
        super(message);
    }
}

public class Main {
    static void consume(Partition log, LoyaltyDb db, Long crashAfter) {
        long pos = log.committed; // a (re)started consumer resumes from the committed offset
        System.out.println("poll from offset " + pos);
        List<LogRecord> batch;
        while (!(batch = log.poll(pos, 3)).isEmpty()) {
            for (LogRecord r : batch) {
                String result = db.apply(r) ? "applied" : "duplicate, skipped";
                System.out.println("  offset " + r.offset() + " " + r.eventId() + " " + result);
                if (crashAfter != null && r.offset() == crashAfter) {
                    throw new CrashException("consumer crashes before committing");
                }
            }
            pos = batch.get(batch.size() - 1).offset() + 1;
            log.committed = pos; // commit only after the whole batch took effect: at least once
            System.out.println("commit offset " + pos);
        }
    }

    public static void main(String[] args) {
        Partition log = new Partition(List.of(
                new LogRecord(0, "evt-a", 101, 10),
                new LogRecord(1, "evt-b", 102, 5),
                new LogRecord(2, "evt-c", 103, 20),
                new LogRecord(3, "evt-b", 102, 5), // the relay crashed and published evt-b again
                new LogRecord(4, "evt-d", 104, 8),
                new LogRecord(5, "evt-e", 105, 12)));
        LoyaltyDb db = new LoyaltyDb();
        try {
            consume(log, db, 4L);
        } catch (CrashException e) {
            System.out.println(e.getMessage());
        }
        System.out.println("restart from committed offset " + log.committed);
        consume(log, db, null);
        StringJoiner totals = new StringJoiner(" ");
        new TreeMap<>(db.points).forEach((k, v) -> totals.add(k + "=" + v));
        System.out.println("points: " + totals + ", events applied " + db.inbox.size());
        // poll from offset 0
        //   offset 0 evt-a applied
        //   offset 1 evt-b applied
        //   offset 2 evt-c applied
        // commit offset 3
        //   offset 3 evt-b duplicate, skipped
        //   offset 4 evt-d applied
        // consumer crashes before committing
        // restart from committed offset 3
        // poll from offset 3
        //   offset 3 evt-b duplicate, skipped
        //   offset 4 evt-d duplicate, skipped
        //   offset 5 evt-e applied
        // commit offset 6
        // points: 101=10 102=5 103=20 104=8 105=12, events applied 5
    }
}

When the side effect is in someone else’s system, the target must cooperate. Payment providers accept an idempotency key, and return the original result for a repeat. An email provider without one leaves a small window, between the send and recording it. In that window, a crash still sends twice. The window can be shrunk, never closed.

Go deeperWhat SQS FIFO's exactly-once means1 min

SQS FIFO queues advertise exactly-once processing. Two mechanisms sit behind it. On the producer side, it drops a message whose deduplication id (or content hash) matches one sent in the last 5 minutes. That removes duplicates from producer retries. On the consumer side, a message group delivers its next message only after the current one is deleted. Neither covers a consumer that charges a card, and then dies before DeleteMessage. Once the visibility timeout expires, the message is delivered again. The consumer still has to be idempotent.

Step 7: Publish reliably from a database write

Checkout must do two things when an order is placed: commit the order to its database and publish an OrderPlaced event. These are two systems with no transaction across them, a dual write, and both orders of operation break:

• Commit, then publish. The process dies between the two (a deploy, an OOM kill). Or the broker is down for a minute. The order exists, but the event never does. No email, no stock reservation, and nothing notices.
• Publish, then commit. The commit fails on a constraint. Downstream now acts on an order that does not exist.

The transactional outbox makes the event part of the database write. Checkout inserts the order and an outbox row describing the event in one local transaction. A separate relay reads unsent outbox rows, publishes them, and marks them sent once the broker acknowledges.

The dual write, and the outbox that fixes it

Scroll or pinch to zoom, drag to pan

• Color key
• Queue or stream
• Database
• Service

A crash before the commit leaves neither the order nor the event. A crash after it leaves a row the relay will publish. A crash between publishing and marking publishes it twice, so the outbox gives at-least-once publishing, and the consumers’ idempotency from step 6 absorbs the repeat. The databases page has the outbox and its relay in code.

The relay is a poller, or a change data capture connector, compared under options. Watch out for one trap. A relay that asks for rows above “the highest id I published” skips rows. Id 102 can commit before id 101. The relay has moved past 101 by the time it commits. Mark rows sent instead, or use CDC, which follows commit order. Prune sent rows after a day or so.

Step 8: Fan-out, delays and priorities

Fan-out. Many consumers each wanting every message is publish-subscribe: one topic, and one independent subscription per consumer with its own position, retries and dead letters.

Broker	Fan-out mechanism
AWS	An SNS topic with one SQS queue subscribed per consumer
RabbitMQ	A fanout or topic exchange, with one queue bound per consumer
Kafka, Kinesis	One consumer group per consumer, all reading the same topic
Google Pub/Sub	One subscription per consumer on a topic

A subscriber that is slow or down falls behind on its own subscription and does not hold up the others.

Delays. SQS delays a message up to 15 minutes (DelaySeconds). A consumer can postpone one by extending its visibility timeout, up to 12 hours. RabbitMQ uses a queue TTL that dead-letters expired messages into the real queue. Kafka has no delays. The usual answer is retry topics with fixed delays (orders-retry-1m, orders-retry-10m). Each is read by a consumer that waits until a message is due. “Send this at 9 a.m. on Friday” belongs in a job scheduler backed by a database.

Retry tiers on a log, so failures do not block the main topic

Scroll or pinch to zoom, drag to pan

• Color key
• Service
• Database
• Cache
• Queue or stream
• Background worker

Moving a failure to a retry topic unblocks its partition. But it also gives up that key’s order. Later events for the same order are processed before the retried one. Use it where order does not matter per key. Or have the consumer park the whole key until the retry succeeds.

Priorities. Give each class of work its own queue and consumer pool, instead of relying on priority flags. Then one-time passwords never wait behind a marketing campaign.

Options and tradeoffs

Work queue or log

Property	Work queue (SQS, RabbitMQ)	Log (Kafka, Kinesis)
After processing	Message deleted	Message kept until retention runs out
Acknowledgment	Per message	One offset per partition: everything before it
A failing message	Retried on its own, others continue, then a DLQ	Blocks its partition unless the consumer moves it aside
Ordering	None (standard), per group (FIFO)	Per partition, so per key
Scaling consumers	Add consumers any time, no limit	Up to one consumer per partition in a group
Several consumers	A queue per consumer, fed by a fan-out topic	A consumer group each, on the same data
Replay	No	Yes, by resetting offsets
Wins for	Jobs and tasks: emails, thumbnails, webhooks, uneven work	Event streams: many readers, replay, ordering, high throughput

TradeoffA queue for tasks, a log for events

We gain

Tasks get per-message retries, dead letters and elastic consumers. Events get one ordered, replayable record that any number of teams can read at their own pace.

We give up

Two kinds of infrastructure to understand and run, and the occasional system that wants both, which often ends as a log feeding a work queue.

Which broker

Option	Model	Ordering	Retention	Operations	When it wins
Amazon SQS	Managed work queue, pull	None, or per group in FIFO	4 days default, 14 max	None: no servers, pay per request	Tasks on AWS. The default background job queue
RabbitMQ	Broker with exchanges and queues, push	Per queue, one consumer	Until acked	A cluster; quorum queues for safety	Rich routing, per-message TTL and priorities, on-premises
Apache Kafka	Partitioned log, pull	Per partition	7 days default, any length	A cluster, or a managed service	Event streams, many consumer groups, replay, 100 MB/s and up
Amazon Kinesis Data Streams	Managed partitioned log (shards)	Per shard	24 hours default, 365 max	Managed; you size shards	Streams on AWS without running Kafka, at moderate volume

TradeoffManaged first

We gain

SQS and managed Kafka remove broker upgrades, disk sizing and failover from the team’s pager. At small and medium scale, that is most of the cost.

We give up

Per-request or per-partition pricing that grows with volume, less control over versions and tuning, and lock-in to one cloud’s semantics.

Go deeperOther brokers you may hear about2 min

• Google Cloud Pub/Sub: managed pub/sub, with per-message acks and ack deadlines (a visibility timeout). Optional ordering keys. And replay, by seeking a subscription to a time.
• Azure Service Bus: a managed work queue with sessions for per-key order, dead-lettering and scheduled messages.
• Redis Streams: a log inside Redis with consumer groups and per-entry acks. Fast and simple, bounded by memory, and only as durable as the Redis setup.
• NATS JetStream: a lightweight log and queue in one binary.
• Apache Pulsar: a log that separates serving brokers from storage (BookKeeper) and supports both stream and queue subscriptions on one topic.
• Kafka share groups (KIP-932, “queues for Kafka”): queue semantics on a Kafka topic, with per-record acks, delivery counts and more consumers than partitions. Early access in Kafka 4.0 and preview in 4.1, so check the current release notes before relying on it.

Delivery guarantee

Option	Risk left	When it wins
At most once (ack first)	Loses a message on every badly timed crash	Telemetry where a gap is fine
At least once + idempotent consumer	Side effects in systems without idempotency keys	The default for anything that matters
Broker transactions (Kafka EOS)	Anything outside the broker	Stream processing from topic to topic

TradeoffAt least once with idempotent consumers, everywhere

We gain

One rule that holds across every broker, every crash and every replay, including replays a person starts on purpose.

We give up

Every consumer needs a stable message id and a dedupe record or conditional write, and the dedupe table needs its own cleanup policy.

Publishing from a database write

Option	How	Lost events	Duplicates	When it wins
Dual write	Commit, then publish	On a crash	On retries	Only where losing an event is acceptable
Outbox, polling relay	Outbox row in the same transaction; a relay polls	None	Yes	Most services: simple, any database
Outbox with CDC	The same row, published from the WAL by Debezium or similar	None	Yes	High volume, low delay, a team that already runs CDC
Listen to yourself	Publish first; the service consumes its own event to write the database	None	Yes	Event-sourced services where the log is the source of truth

TradeoffThe transactional outbox

We gain

Every committed change produces its event, even through crashes, deploys and broker outages, because the broker is no longer on the commit path.

We give up

A relay to run, 100 ms to 1 s of added delay with polling, an outbox table to prune, and duplicates that consumers must absorb.

Choosing a partition count

A log’s partition count caps both throughput and consumer parallelism, and changing it later remaps keys. Size it from three numbers:

1. Write throughput. Target 50 MB/s, and a partition sustains about 10 MB/s of produce in your own tests: at least 5 partitions.
2. Consumer throughput. This is usually the binding one. A consumer that writes each event to a database processes 2 MB/s per partition: 25 partitions.
3. Growth. Adding partitions breaks per-key order during the change, so provision for one to two years: 25 today and 2× growth gives 48 or 50.

Slow work changes the answer. 2,000 messages a second at 200 ms each keep 400 in flight (Little’s law). So strictly ordered consumers would need 400 partitions. Some consumers process different keys of one partition in parallel, each key still in order (Confluent’s parallel consumer). That does the same job with 24 partitions and 20 workers each.

Go deeperToo few or too many partitions, and Kinesis shards1 min

Too few partitions	Too many partitions
Consumer parallelism capped below what the load needs	More files, memory and replication traffic per broker
A hot key shares its partition with more neighbors	Longer leader elections and rebalances; more open connections
Adding them later reorders keys and moves state	Small batches per partition, so lower throughput and higher cost per byte

Kinesis makes the same choice in shards. Each takes 1 MB/s or 1,000 records a second in, and 2 MB/s out. Resharding splits or merges shards.

TradeoffPartitions for two years of growth

We gain

No repartitioning, with its broken key order and moved consumer state, for the life of most designs.

We give up

Paying for partitions before the traffic arrives, and smaller batches until it does.

How it fails

Failure	What breaks	How you see it	Fix
Poison message	A consumer crash-loops; on a log, its partition stops	One message’s receive count climbing; one partition’s lag growing alone	A retry limit, a DLQ, and catching errors per message rather than per batch
Consumers slower than producers	The backlog grows without limit, so delay grows without limit	Lag in time, or age of the oldest message, rising steadily	More consumers (up to the partition count), batching, or fix the slow dependency
Retention runs out first	A log deletes old segments, a queue deletes old messages: silent data loss	Lag in time approaching retention	Alert at half the retention; longer retention or tiered storage
Visibility timeout below processing time	A healthy slow consumer’s message is given to a second one	Receive count above 1 with no errors logged	Timeout at several times p99, heartbeats for long work
Rebalance storm	A consumer that takes longer than max.poll.interval.ms (5 min) is evicted, its partitions move, and the new owner repeats the slow batch	Rebalances per hour; lag spikes on every deploy	Smaller poll batches, cooperative or KIP-848 rebalancing, static membership
Hot partition	One key’s traffic exceeds what one consumer handles	One partition’s lag rising while the others are flat	Split the key if its order allows; make that consumer faster
Unwatched DLQ	Failures stop being visible; customers simply never get their email	DLQ depth growing for days	Alert on any message in the DLQ, give it an owner and a redrive tool

Go deeperThree more ways it fails1 min

Failure	What breaks	How you see it	Fix
Retry storm	A downstream outage makes every consumer retry at once, prolonging it	Error rate and request rate to the dependency rising together	Exponential backoff with jitter, a circuit breaker, pause consumption
Relay or CDC stopped	Orders commit, events do not flow	Age of the oldest unsent outbox row; replication slot size	Alarm at 30 s; run the relay with a standby
Non-idempotent consumer	Double charges and double emails on every crash, deploy and replay	Duplicate effects with the same message id	A dedupe key in the same transaction as the effect

The 2,000-order example drawn whole, with the outbox, the log and three consumer groups. Break any part and see what the shopper notices:

Order events through a queue, then a log · architecture

Break it: what fails, who notices, what users see

Color keyServiceDatabaseQueue or streamBackground workerNew in this stageAsync or occasional

Break

An email workerThe outbox relayThe Kafka clusterThe email providerThe dead-letter topicThe orders databaseAn order service instance

Failure: click any box to break it

The Kafka cluster goes down

What breaks

The relay cannot publish.

Consumers have nothing new to read.

How we notice

Produce errors and under-replicated partitions on the relay and broker dashboards.

What users see

Nothing on checkout: the outbox absorbs the backlog.

Downstream work waits, then drains at full speed.

What we do

Replication factor 3 with min.insync.replicas 2 survives one broker.

A full outage is a growing outbox table, so size its disk for hours of orders.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

Common mistake

"With a queue in front, the slow service can no longer hurt us"

A queue converts overload into delay, not into success. Say consumers process 1,800 messages a second, and producers send 2,000. The backlog grows by 720,000 an hour. Every message waits longer than the last. Eventually the queue hits its retention or its size limit, and loses data. A queue needs consumers sized for the sustained rate, and an alarm on the age of the oldest message. And it needs a decision about what to shed when the backlog gets too old to be useful.

At scale

Numbers worth knowing:

Quantity	Typical value
SQS send or receive latency	Tens of ms
SQS standard queue throughput	Nearly unlimited; scales on its own
SQS FIFO throughput	300 API calls/s per action, 3,000 messages/s with batches of 10; far more in high-throughput mode
SQS limits	Visibility 30 s default, 12 h max. Retention 4 days default, 14 max. Delay 15 min max. Long poll 20 s
Kafka partition	Tens of MB/s of produce
Kafka end-to-end latency, one region, acks=all	About 5 to 20 ms at moderate load
Kafka consumer defaults	Session timeout 45 s, max.poll.interval.ms 5 min, max.poll.records 500
Kinesis shard	1 MB/s or 1,000 records/s in, 2 MB/s out
Largest message, by default	About 1 MB in Kafka, 1 MiB in SQS (256 KiB before 2025). Larger payloads go to blob storage

At 10× (20,000 messages a second), work queues mostly keep working. SQS scales, and consumers are added by autoscaling on queue depth. A log now needs its partition count to have been right. At a few ms of processing per message, a strictly ordered consumer handles a few hundred messages a second per partition. So 20,000 needs about 64 partitions, or parallel processing within each. Dedupe tables grow by 1.7 billion rows a day. They need time-based partitions, dropped after the longest possible redelivery window, a few days.

At 100× (200,000 messages a second, about 200 MB/s at 1 KB each), one Kafka cluster still copes, on 6 to 12 brokers. But costs and blast radius dominate. With replication factor 3 across zones, every byte written crosses zones twice. Cross-zone transfer, at about $0.01 per GB each way, becomes one of the largest lines on the bill. Consumers fetching from a replica in their own zone cut the read side. Clusters split by domain, so one runaway producer cannot slow everyone. And old segments move to object storage (tiered storage).

What to monitor:

• Lag in time per consumer group and partition, or the age of the oldest message on a queue. 10,000 messages behind is nothing at 50,000 a second and an hour at 3 a second.
• DLQ depth, alarmed on any message, with the error that sent it there.
• Redelivery rate (receive counts above 1), error rate and rebalances per consumer group.
• Produce errors and latency, and under-replicated partitions on the brokers.
• Outbox health: age of the oldest unsent row, CDC slot size.
• Retention headroom: the worst lag in any group against the topic’s retention.

In the interview

Interviewer asksProduct says every order confirmation email must go out exactly once. What do you build?

Exactly-once delivery is not available from any broker, because an ack can be lost, so I build at least once with an idempotent consumer. The order and an outbox row commit in one transaction, a relay publishes to the queue, and the email worker acks only after sending. The worker records the event id in a sent-emails table and skips ids it has seen. The one remaining window is a crash between the provider accepting the email and the worker recording it. If the provider takes an idempotency key, I pass the event id and the window closes. Otherwise the contract I offer product is a rare duplicate, never a lost email.

Interviewer asksConsumer lag on the billing topic keeps growing. Walk me through it.

First, all partitions, or one? One partition growing alone is a hot key or a poison message. Check per-key rates, and whether the committed offset is stuck on one record. All partitions growing means consumers are slower than producers. Usually that’s because of a slow database they call. If consumers are fewer than partitions, add them. If equal, adding more does nothing. So batch the database writes, or process keys in parallel within a partition. Also check rebalances. A consumer evicted for exceeding the poll interval repeats its batch. Meanwhile, I compare lag in time with retention, because that is when lag turns into data loss.

Common mistake

"Kafka has exactly-once, so duplicates are solved"

Kafka’s exactly-once covers writes back into Kafka: a transaction commits a stream processor’s output records and input offsets together. A consumer that charges a card, sends an email or writes to Postgres is outside that transaction and repeats the effect after a crash. Those consumers still need a dedupe key in the target system or offsets stored with their own writes.

Mid-level covers

Puts slow work behind a queue. Knows consumers ack after processing, so messages can be redelivered. Adds a dead-letter queue. Knows Kafka keeps order per partition.

Senior adds

States delivery guarantees as effects after failures. Makes consumers idempotent, in the same transaction as their writes. Publishes through an outbox. Picks a queue or a log by whether the data is a task or an event. Sizes partitions from consumer throughput. Monitors lag in time, against retention.

Seen in

• Message queue: the broker itself, from partitions to replication.
• Payment system: an outbox in the ledger transaction, events keyed by payment id.
• Notification system: Kafka with acks=all after dedupe, delay topics for retries, a lane per channel.
• News feed: post events keyed by author drive fan-out workers, redelivered when an offset was not committed.
• Chat: events partitioned by conversation for ordered fan-out, with order itself from sequence numbers.
• Web crawler: small pointers on Kafka, pages in WARC files.
• Ad click aggregator: clicks acknowledged by two zones before the redirect, then replayable.
• Top-K: views keyed by item, with Kafka offsets checkpointed alongside the counts.
• Job scheduler: at-least-once runs, a dead-letter state, and an outbox streamed by CDC.
• Instagram: a media queue, with video on its own queue so photos never wait.
• Video streaming: a job queue of wake-ups over a task table, and views on Kafka.
• Ticket booking, hotel reservation and ride hailing: an outbox, so a broker outage never blocks a booking or a trip.
• Leaderboard: an outbox and absolute totals, so repeats are harmless.
• Proximity service and distributed cache: CDC into Kafka keeps an index or a cache in step with the database.
• Typeahead and URL shortener: query and click logs on Kafka, off the request path.
• Stock exchange: the counter-example, no broker on the order path, Kafka only for clearing.

Recap

Order events through a queue, then a log · architecture

The full design, stage by stage

Grow itTrace a request

Color keyServiceDatabaseQueue or streamBackground workerNew in this stageAsync or occasional

Play

PreviousNext

Stage

123

Stage 3 of 3

A log for fan-out and replay

Why this stage

Inventory and analytics want the same order events, and analytics needs to reprocess last week after fixing a bug.

What we add

The relay publishes to a Kafka topic keyed by order id: 12 partitions, 7 days of retention.

Each team is its own consumer group with its own offsets, so each reads every event at its own pace.

What it costs

Partitions sized up front, order kept only per order id, a cluster to run or pay for, and every consumer group must be idempotent.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.The full design. Step back through the stages and say what forced each one, then follow an order and a crash.

Why. Decoupling, buffering bursts, retries. Paid for with delay, failures that surface elsewhere, and duplicates. Anything the response depends on stays synchronous.

Work queue. Receive hides the message, ack deletes it, no ack means redelivery after the visibility timeout. A receive limit moves poison messages to a DLQ. Redelivery reorders.

Log. Messages kept for retention, offsets per partition per consumer group, replay by resetting offsets, at most one consumer per partition in a group.

Order. Per key, through partitions or FIFO groups. Costs head-of-line blocking, a parallelism cap and hot keys.

Guarantees. Ack first: at most once. Ack after: at least once. Plus an idempotent consumer: effectively once. Exactly-once delivery does not exist. Exactly-once effects do.

Outbox. Event row in the same transaction as the change, published by a relay or CDC. No lost events, some duplicates.

Numbers. SQS visibility 30 s default, 12 h max, retention up to 14 days. Kafka retention 7 days default. Kinesis shard 1 MB/s in, 2 MB/s out. Lag measured in time.

Further reading

• On this site: Design a message queue, for the broker’s internals. Distributed transactions, for sagas built on events. Partitioning, for how keys map to partitions. And replication, for the change streams that CDC reads.
• Martin Kleppmann, Designing Data-Intensive Applications, chapter 11 (Stream Processing): logs versus queues, and exactly-once in practice.
• Apache Kafka documentation, Design: delivery semantics, the idempotent producer and consumer positions, from the source.
• Chris Richardson, Pattern: Transactional outbox: the pattern, its relay variants and related patterns.