Kafka or RabbitMQ? What the Message's Lifecycle Changes

Comparisons between Kafka and RabbitMQ usually open with a feature table: throughput, replay, routing, ordering, retry. The table is useful, but on its own it does not make the choice much easier, because the rows look unrelated to each other.

I find it more useful to read those differences from somewhere else:

What happens to a message once it enters the broker?

RabbitMQ tracks the delivery state of a message and removes it from the queue once the ack arrives. Kafka keeps the message in a log; where you got to is something the consumer carries, in its own offset.

Most of the meaningful differences between the two start right there: the lifecycle of a message inside the broker, who holds the consumer’s position, and what the broker does after a message has been processed. Replay, consumer groups, retry and routing all follow from it.

First: why do we need a broker at all?

A message broker is the intermediate layer that makes communication between two applications asynchronous. The producer drops a message off at the broker; the consumer picks it up when it is ready. Neither side has to know the other, or be up at the same time.

Consider a system where the order service calls the email, invoice, stock and shipping services directly over HTTP:

@Transactional
public Order createOrder(OrderRequest request) {
    Order order = orderRepository.save(Order.from(request));

    emailService.sendConfirmation(order);      // if it is slow, the order is slow
    invoiceService.create(order);               // if it fails, the order rolls back
    stockService.reserve(order);
    shippingService.schedule(order);

    return order;
}

@Transactional
public Order createOrder(OrderRequest request) {
    Order order = orderRepository.save(Order.from(request));

    // The order service does not know who is listening
    broker.publish(new OrderCreated(order.getId(), order.getTotal()));

    return order;
}

The main benefits of using a broker come down to a handful of things:

Once you have decided you need a broker, the real decision starts: which one?

Two models: a post office and an archive

The same thing as a flow:

graph LR
    P1["Producer"] --> EX{{"Exchange"}}
    EX -->|"routing key"| Q1[("Queue")]
    Q1 -->|"push"| C1["Consumer"]
    C1 -.->|"ack → message leaves the queue"| Q1

In RabbitMQ the flow is one-way and finite: the broker pushes the message to a consumer and removes it from the queue when the ack arrives. The only thing coming back is delivery information, not the message itself.

graph LR
    P2["Producer"] --> T["Topic partition<br/>append-only log"]
    T -->|"offset 512"| C2["Consumer A"]
    T -->|"offset 40"| C3["Consumer B"]
    T -->|"from offset 0"| C4["Consumer C<br/>just joined"]

In Kafka the log sits in the middle and each consumer reads from its own position. Consumer B is behind, Consumer C just connected and is reading history from the start. Neither affects the other, and nothing touches the log.

The sections that follow look at what these two models lead to.

When the message stays in the log: replay

If a message is still there after being read, you can go back and read it again. That is where Kafka’s replay comes from.

Four Kafka concepts are enough to see how it works.

Topic and partition

A topic is the logical category messages are written to: orders, user-signups, payment-events.

A topic is split into partitions. Each partition is an ordered log file that is only ever appended to:

orders topic, 3 partitions

partition-0:  [0][1][2][3][4][5] ← new messages go here
partition-1:  [0][1][2][3]
partition-2:  [0][1][2][3][4][5][6][7]

A partition is not just a storage detail — it is also Kafka’s unit of parallelism. We will get to what that implies shortly.

Which partition a message goes to is decided by the partitioner. By default:

• If the message has a key: a partition is chosen from the key’s hash; the same key always lands on the same partition.
• If there is no key: messages are spread evenly across partitions.

Offset

The sequence number of each message within a partition. The consumer commits “I have read up to 4523” as an offset.

Replay is nothing more than that: rewind the offset and the same messages flow again.

current state:      offset = 4523
rewind the offset:  offset = 4100
result:             423 messages are reprocessed

If a bug in the email service sent the wrong content for the last two hours, fixing the code and rewinding the offset is enough. In RabbitMQ those messages left the queue the moment they were acked.

Consumer group

Consumers sharing a group.id form a group, and there are two rules:

graph LR
    T["orders topic<br/>3 partitions"]

    T --> G1["Group: invoice-service<br/>offset 4523"]
    T --> G2["Group: analytics<br/>offset 4523"]
    T --> G3["Group: search-index<br/>offset 120 — behind"]

This has a practical consequence: if you need to rebuild the search index from scratch tomorrow, you open a new consumer group and rewind its offset to the beginning. The history in the topic feeds the new service, and none of the existing services have to do anything.

Retention

Since messages are not deleted after being read, they have to live somewhere. The retention policy governs that: time-based (say 7 days), size-based, or via log compaction — where only the latest value for each key is kept.

When the broker tracks delivery: being able to reject a message

Now look the other way. In RabbitMQ, when a message is delivered to a consumer the broker knows about it and counts the message as “in flight”. That knowledge enables something Kafka has no direct equivalent for: rejecting a message.

Ack, nack, requeue

The consumer sends ack once it has processed the message, and the message leaves the queue. If it cannot process it, nack either puts it back on the queue (requeue) or drops it.

@RabbitListener(queues = "email-jobs")
public void handle(EmailJob job, Channel channel,
                   @Header(AmqpHeaders.DELIVERY_TAG) long tag) throws IOException {
    try {
        emailSender.send(job);
        channel.basicAck(tag, false);              // processed
    } catch (TransientFailure e) {
        channel.basicNack(tag, false, true);       // transient, put it back
    } catch (PermanentFailure e) {
        channel.basicNack(tag, false, false);      // permanent, route to the DLX
    }
}

There is no direct equivalent of those three lines in Kafka. There is no per-message rejection mechanism there — there is an offset:

• If you commit the failed message’s offset, that message is skipped.
• If you do not commit it, that partition does not advance and everything behind it waits.

The second is called head-of-line blocking, and it is one of the most common difficulties in task queue scenarios.

Dead Letter Exchange

Messages that are rejected, that hit their TTL, or that overflow a queue limit get routed to a DLX. This is the built-in way to quarantine bad messages and to build delayed retries:

graph LR
    Q["email-jobs"] -->|"nack (requeue=false)"| DLX{{"email-dlx"}}
    DLX --> DLQ[("email-dlq<br/>quarantine")]
    DLQ -.->|"inspect, fix,<br/>resend by hand"| Q

RabbitMQ offers this out of the box, whereas in Kafka you generally have to design retry topics and a separate DLQ flow yourself:

email-jobs → (failure) → email-retry-5m → (failure) → email-retry-30m → (failure) → email-dlq

A common pattern that works well — but an extra layer you have to write and operate.

Prefetch

Sets how many messages the broker will send a consumer at once without waiting for an ack:

spring:
  rabbitmq:
    listener:
      simple:
        prefetch: 1
        acknowledge-mode: manual

Keeping prefetch low pays off when job durations are unpredictable. At a high value, the jobs handed to a worker up front wait alongside it whenever that worker gets stuck on a slow job; at a low value, each worker takes the next job as it becomes free.

Who does the routing?

If a message leaves the queue after its ack, getting it to the right queue is the broker’s responsibility. If the message stays in a log, who reads what is the reader’s decision.

That is why the routing models of the two systems look nothing alike.

RabbitMQ: exchanges

The producer sends the message to an exchange, not directly to a queue. The exchange decides which queues to copy it to, based on binding rules and the message’s routing key.

Producer → Exchange → (Binding + Routing Key) → Queue → Consumer

There are four exchange types:

In a topic exchange, * means one word and # means zero or more words. The pattern order.# also catches multi-level keys like order.eu.created.

RabbitMQ offers a more flexible routing model here: which queues a message ends up in is determined by broker configuration, without changing producer or consumer code.

Kafka: topic, partition, key

Kafka has no equivalent of RabbitMQ’s exchange/binding model. The producer writes to a topic, and the partition is chosen via the key and the partitioner. The consumer knows which topic it wants to read.

If you need filtering, there are two routes: split the subjects into separate topics, or skip the messages you do not care about on the consumer side. Neither fully replaces the flexibility exchanges give you.

What limits parallelism?

Since the partition is Kafka’s unit of parallelism, a consumer group cannot have more active consumers than there are partitions.

orders topic, 4 partitions, one consumer group

4 consumers  → each takes 1 partition        ✅
6 consumers  → 4 work, 2 sit idle            ⚠️

RabbitMQ has no such partition-derived limit on a consumer group; you can attach as many consumers to a queue as you like. Throughput is of course still bounded by broker, network and consumer resources — but that bound is a capacity question, not a structural ceiling.

Kafka:     consumer group parallelism ≤ partition count
RabbitMQ:  a queue's consumer count is not tied to a partition count

Ordering

Kafka’s ordering guarantee is within a partition, not across a topic. That distinction matters, because the real requirement is usually not “process all events in order” but “process one entity’s events in order”.

The key mechanism covers exactly that:

// key = orderId → all events for this order land on the same partition, in order
kafkaTemplate.send("order-events", order.getId(), new OrderCreated(...));
kafkaTemplate.send("order-events", order.getId(), new OrderPaid(...));
kafkaTemplate.send("order-events", order.getId(), new OrderShipped(...));

graph LR
    A["orderId = 1042"] --> P1["partition-2<br/>created → paid → shipped"]
    B["orderId = 1043"] --> P2["partition-0<br/>created → paid"]
    C["orderId = 1044"] --> P1

If you need global ordering across a topic, you need a single partition — which removes parallelism entirely and is rarely the right trade.

On the RabbitMQ side, a single queue with a single consumer preserves FIFO order. Once you attach several consumers to scale, messages spread across workers and processing order is no longer guaranteed. Requeue affects order too: a message put back with nack returns not to the tail of the queue but as close to its original position as possible — so it is redelivered almost immediately.

If you want behaviour similar to Kafka’s key→partition model, the Consistent Hash Exchange plugin can distribute messages into separate queues by key — but it is not as built-in as it is in Kafka.

If you have an entity-level ordering requirement — event sourcing, state machines, an order lifecycle — Kafka’s model answers it more directly.

What is the same in both: delivery guarantees

So far we have looked at differences. Delivery guarantees, though, are defined at the same three levels in both systems:

In Kafka, on the producer side acks=all together with min.insync.replicas waits for the message to be written to enough replicas; enable.idempotence=true stops producer retries from creating duplicates — and since Kafka 3.0 that setting is on by default, so in most setups it is not something you switch on but something you avoid switching off. On the consumer side the guarantee is decided by when you commit the offset:

// Process first, then commit
process(record);
consumer.commitSync();

// If it crashes after processing but before the commit
// → the same message arrives again

// Commit first, then process
consumer.commitSync();
process(record);

// If it crashes after the commit but before processing
// → the message is never processed

In RabbitMQ, on the producer side publisher confirms verify that the broker received the message; making the message persistent and the queue durable gets it written to disk. On the consumer side manual ack means at-least-once and auto-ack means at-most-once.

Kafka’s transactions feature offers exactly-once, but its scope is essentially “read from Kafka → process → write to Kafka” flows. When you write to an external system — a database, an HTTP API, an email provider — that guarantee does not hold. RabbitMQ makes no exactly-once claim in the first place.

if (!processedEvents.markIfAbsent(event.id())) {
    return;   // already processed
}
process(event);

When does the consumer actually get the message?

So far we have looked at what happens to a message inside the broker. But when does it cross over to the consumer? When there is spare CPU? When memory frees up?

Neither. The broker knows nothing about your CPU or your memory. Both systems bound the flow with something else entirely — and that is exactly where the answer to a commonly asked question lives: if a flood of events arrives at once, does the application fall over?

Kafka: the consumer asks, the broker answers

Kafka is a pull model. The broker pushes nothing to anyone; the consumer calls poll() on a loop and processes however many records that call returned.

while (running) {
    ConsumerRecords<String, Report> records = consumer.poll(Duration.ofMillis(500));

    for (ConsumerRecord<String, Report> record : records) {
        process(record);          // however long this takes, the loop turns that slowly
    }

    consumer.commitSync();
}

The only thing setting the pace of that loop is how fast process() is. If it is slow the loop turns slowly, poll() is called less often, and fewer records come off the broker. In other words, backpressure in Kafka comes from the model itself: nothing arrives unless you ask for it.

A handful of settings decide how much one poll() brings back:

In practice the only one you usually tune is max.poll.records. If a job takes 30 seconds, there is no point pulling 500 of them:

spring:
  kafka:
    consumer:
      max-poll-records: 1     # one poll = one job

RabbitMQ: the broker pushes, prefetch is the brake

RabbitMQ is a push model. The moment you subscribe with basic.consume, the broker starts streaming messages at you — as they arrive, without you asking.

And the danger there is real. Without a brake the broker sends everything in the queue as fast as the connection allows, and the client library buffers all of it in memory. That is the classic route to an OutOfMemoryError.

The brake is the prefetch setting we touched on earlier:

prefetch = 10

broker → sends the consumer at most 10 UNACKED messages
consumer acks 1 → the broker sends 1 more

So prefetch is a credit-based sliding window: when the number of unprocessed messages in your hands hits the ceiling, the broker goes quiet.

Picking the value comes down to a simple rule:

short jobs of similar duration  → high prefetch (100–250), amortises the round trip
long jobs of variable duration  → prefetch: 1, each worker takes the next one as it frees up

The two side by side

The answer to the question this section opened with is that last row:

In Kafka you have to work at it to drown your application. In RabbitMQ you have to have set prefetch in order not to.

Long-running work: the 30-minute report

Now to the concrete question: a report takes 30 minutes, and we ack the message once the work is done. Is that safe?

In RabbitMQ, largely yes — with one limit. In Kafka, no — and on default settings the outcome is not a quiet failure but a very loud one: the same report is generated over and over.

What happens in Kafka

There are two separate timeouts, and they get confused constantly:

Heartbeats are sent by a separate background thread, so a 30-minute job does not trip session.timeout.ms; the process keeps looking alive. It is the second one that gets you:

00:00  poll() → 1 record: "monthly reconciliation report"
00:00  report generation starts
05:00  max.poll.interval.ms exceeded
       → the broker declares the consumer dead and evicts it from the group
       → rebalance: the partition moves to another worker
       → the new worker starts from the last committed offset
       → it begins generating THE SAME REPORT from scratch
30:00  the first worker finishes and calls commitSync()
       → CommitFailedException: it no longer owns that partition
       → 30 minutes of work thrown away, the offset never moved
35:00  the second worker was evicted at its own 5-minute mark... and round it goes

So yes: on default settings Kafka will generate the same report forever and never commit it. Worse, because the report really is being generated each time, the system looks busy and the problem takes a while to notice.

What happens in RabbitMQ

RabbitMQ has no equivalent of max.poll.interval.ms; you can hold a message unacked for as long as you like. The connection heartbeat (60 s by default) is sent by a separate I/O thread, so long processing does not disturb it.

But there is a limit, and it lands right on this scenario:

# 2 hours
consumer_timeout = 7200000

Side by side:

So how should it be designed?

The solution that first comes to mind — “what if we ack on arrival and process asynchronously in the background?” — is intuitively pointed in the right direction, but as stated it breaks two things at once. Here are four approaches side by side:

1. Raising the timeout

The simplest solution, and usually sufficient:

spring:
  kafka:
    consumer:
      max-poll-records: 1          # one poll = one report
      properties:
        max.poll.interval.ms: 2700000   # 45 minutes

The price: a worker that genuinely crashes now leaves its partition unowned for 45 minutes. You have traded fast failure detection for long job durations. If the job has a known upper bound, that is a perfectly reasonable trade.

2. pause() + async + empty poll()

Kafka’s own mechanism for this. You hand the work to a background thread but keep calling poll() — after pausing the partition first, so no new records arrive:

ConsumerRecords<String, Report> records = consumer.poll(Duration.ofMillis(500));

for (ConsumerRecord<String, Report> record : records) {
    consumer.pause(consumer.assignment());              // ask for no new records
    Future<?> job = executor.submit(() -> generate(record.value()));

    while (!job.isDone()) {
        consumer.poll(Duration.ofSeconds(1));           // returns nothing, but resets the clock
    }

    consumer.commitSync();                              // the work really is done
    consumer.resume(consumer.assignment());
}

poll() on a paused consumer returns no records but keeps both the heartbeat and the max.poll.interval.ms clock alive. So the job can take 30 minutes without the consumer being evicted — and if it genuinely crashes, that is still detected in seconds.

With Spring Kafka you can get the same behaviour through the container’s pause() / resume() methods; but if the duration is predictable, approach 1 costs a lot less code.

3. Acking immediately and processing behind — why it is a trap

This creates two separate problems at once:

So: if losing the report is not acceptable, do not take this route.

4. A job table: reducing the message to a trigger

For work that runs for hours, reports progress and can be cancelled, the right answer is usually this: take the work’s durability away from the broker and put it in your own database. The message is no longer the work — it is only a trigger.

graph TD
    M["report-jobs message<br/>carries only a job_id"] --> C["Consumer"]
    C --> CL["move the report_jobs row<br/>to RUNNING<br/>(idempotent claim + lease)"]
    CL --> OK["commit the offset IMMEDIATELY"]
    OK --> W["run the work in the background,<br/>write progress to the row"]
    W --> D["DONE"]
    W --> F["FAILED + attempt++"]
    RP["Reaper<br/>RUNNING rows with an expired lease"] -.->|"re-enqueue"| M

The key is the third box: the offset is committed immediately, which looks like approach 3 — but it is not at-most-once. Because durability no longer lives in the offset, it lives in the database row. The broker’s job has shrunk to “deliver the trigger once”; you are the one keeping the record of whether the work actually happened.

What you get in return:

• You can show the job’s status and progress to the user
• Attempt count, error message and duration sit somewhere queryable
• Stuck jobs get picked up by the reaper once their lease expires
• Cancelling is just moving the row to CANCELLED

The rule for long-running work

Holding a long job in Kafka takes a setting — max.poll.interval.ms defaults to 5 minutes, and exceeding it does not give you a silent loss but an infinite repeat loop. Holding one in RabbitMQ is more natural, but you need to know about consumer_timeout’s 30-minute default. For work running beyond a few minutes, the right answer is the same on both: reduce the message to a trigger and keep the work’s state in your own database.

The real distinction: is the message a job or a fact?

So far we have looked at how the two models differ. Now the question that actually matters: with a concrete requirement in front of you, which one do you pick?

A feature list does not answer that on its own, because both can do most of the rows. A more discriminating question is this:

What is this message saying: a job that needs doing, or a fact that has already happened?

"Generate this report"    → job   — it must be done, and once done it is worthless
"Order 1042 was created"  → fact  — true no matter who reads it; deleting it loses information

That distinction decides almost everything else:

The right column is the world Kafka describes; the left column is RabbitMQ’s. The row that stands out most is the third:

Kafka’s greatest strength — the message staying put and being re-readable — is not a win on the task queue side. It is a risk you have to manage. An accidentally rewound offset means “rebuild the projection” in an event stream; in an email queue it means a second email to 40,000 people.

None of that is convincing while it stays abstract. So let us build the same workload on both brokers and see where each one struggles.

What if we built report generation on Kafka?

A concrete scenario:

Work         : reports requested from the customer portal
Volume       : ~12,000 requests a day
Duration     : 2 seconds (daily summary) – 8 minutes (monthly reconciliation)
Month-end    : ~20x the normal request rate for three days
Requirements : failed reports must be retried, enterprise customers go first

A Kafka setup looks perfectly reasonable: a report-jobs topic, 12 partitions, 12 workers in a report-workers consumer group. And it works — there is nothing in this scenario Kafka cannot do. The problems do not show up on day one; they show up in month three.

1. A long job holds up everything behind it

An 8-minute monthly reconciliation lands on partition-7. Within the group only one worker consumes that partition, and it receives its messages in order. The 40 two-second reports behind it wait 8 minutes — while the other 11 workers sit idle.

graph LR
    subgraph K["Kafka — assignment happens at write time"]
        KP7["partition-7<br/>8 min · 2 s · 2 s · 2 s"] --> KW7["worker-7<br/>busy, the queue does not move"]
        KP3["partition-3<br/>empty"] --> KW3["worker-3<br/>idle"]
    end

graph LR
    subgraph R["RabbitMQ — assignment happens at delivery time"]
        RQ[("report-jobs<br/>8 min · 2 s · 2 s · 2 s")] --> RW1["worker-1<br/>took the 8-minute job"]
        RQ --> RW2["worker-2<br/>took the next one"]
        RQ --> RW3["worker-3<br/>took the next one"]
    end

The difference fits in one line, and that line explains most of the task queue argument:

Kafka:     message → partition → that partition's worker   (assigned at write time)
RabbitMQ:  message → queue     → whichever worker is free  (assigned at delivery time)

If job durations are similar, you never feel this. If they vary — and in report, PDF and image-processing workloads they typically do — it shows up directly as queueing delay.

2. A report that fails

A report times out while uploading to S3. In Kafka you have two options, and both are uncomfortable:

• If you do not commit the offset, the message comes back — but that partition does not advance. If the failure is permanent (a bad parameter, a deleted customer) the partition is stuck on that message forever and everything behind it stops. This is the poison message problem.
• If you commit and write the message to a retry topic, the job gets unblocked, but Kafka has no “deliver this in five minutes”. The retry topic’s consumer has to look at the message timestamp and wait — which means deliberately stalling its own partition with pause()/resume().

The layer you end up writing looks like this:

report-jobs → report-retry-5m → report-retry-30m → report-retry-2h → report-dlq

  + a separate consumer per level
  + pause/resume logic that waits on the message timestamp
  + carrying the original topic, partition, offset and error in headers
  + a tool to selectively resend from the DLQ
  + four more consumer lag metrics to watch

In RabbitMQ the equivalent behaviour comes out of the box:

channel.basicNack(tag, false, false);   // → DLX → delay queue → retry

3. Month-end: twenty times the load

The backlog is growing and you want more workers.

Kafka
  12 partitions, one consumer group
  deploy 40 workers  →  12 run, 28 sit idle

  Raising partitions from 12 → 48:
    - if you use keys, the same key lands elsewhere and ordering breaks during the change
    - reducing it again later is not supported
    - so this is not a decision to make in a hurry at month-end

RabbitMQ
  replicas: 12 → 40
  done

In Kafka the partition count is a capacity ceiling, chosen up front and permanently. In task queues the load is usually the thing you cannot predict, which puts that ceiling in an awkward place: too many partitions means needless overhead, too few means a scaling cap.

4. Putting enterprise customers first

Kafka’s log is ordered; “move this message to the front” is not a concept there. What you can do is open a separate report-jobs-priority topic and have the worker read the two topics with weighting — which means you write the priority logic, the fair share between the two topics, and the starvation protection.

In RabbitMQ this is a queue argument:

// into a queue declared with x-max-priority
rabbitTemplate.convertAndSend("report-jobs", job, m -> {
    m.getMessageProperties().setPriority(customer.isEnterprise() ? 9 : 1);
    return m;
});

5. “Generate this report at 3 a.m.”

Delayed delivery is not built into Kafka. Receiving the message early and holding it in the application blocks the partition, so in practice the work moves to a scheduler or a database table.

RabbitMQ has two built-in routes: put a TTL on the message and let the DLX drop it into the target queue, or use the delayed message plugin.

So what would picking Kafka have gained?

To keep the list from being one-sided, here is the other direction — what Kafka genuinely brings to this scenario:

Summary: the answer to “why RabbitMQ”

Look at those gains: all of them are real, and none of them is a daily need in the report-generation scenario. The volume is low, there is one consumer, and if the history has analytical value you are already getting it from the database. What you give up, on the other hand — retry, priority, delay, elastic worker scaling, long jobs not blocking the ones behind them — is exactly what this scenario runs into every day.

So the answer is this:

The reason to pick RabbitMQ is not that Kafka cannot do the job. It is that in this scenario you do not need the things Kafka is strong at, and the things it is weak at are precisely your daily requirements.

Turn that sentence around and you get the reason to pick Kafka — which is what the next section is for.

The other direction: what if we built an event stream on RabbitMQ?

This time the scenario is: an OrderCreated event is published and five services consume it — invoicing, stock, email, analytics and loyalty points.

Building that in RabbitMQ is not hard: one fanout exchange and five durable queues bound to it. And it works fine for months. The difficulty appears as the system ages.

1. A sixth consumer arrives

A search index is added. It sees the messages that arrive after its queue is bound; the two years of order history are not there.

What you have to do: a separate backfill script that scans history out of the database. And that script brings its own problems — where the backfill overlaps the live stream, the same order can be processed twice or the ones right at the cutover can be missed; and it is a second code path, different from and less tested than the real one in production.

In Kafka the same job is three lines of configuration:

spring:
  kafka:
    consumer:
      group-id: search-indexer     # a new group → its own offset
      auto-offset-reset: earliest  # read from the start of the log

2. A consumer ran broken for six hours

The invoice service was deployed with a wrong VAT calculation and spent six hours processing and acking messages. The messages left the queue; there is no getting them back. Fixing it comes down to a database scan again.

In Kafka this is just rewinding the consumer group’s offset by six hours — the messages are still in the log.

3. One lagging consumer can slow down the whole broker

The analytics service was down for three hours and millions of messages piled up in its queue. RabbitMQ tries to keep messages in memory first; once the memory threshold is crossed the broker applies flow control, and that slows down not just that queue but the publishers. In other words, the analytics service’s problem reflects back onto the order service.

In Kafka there is nothing special about a lagging consumer: the messages are already on disk and it simply reads from an older offset. Reads that fall outside the page cache create disk I/O, but that does not reach the producers.

RabbitMQ:  a backlog is the broker's problem   → it can spread
Kafka:     a backlog is the consumer's problem → it stays contained

This is one of the least discussed differences between the two models, and one of the most felt in operations.

4. One copy per consumer

A fanout exchange copies the message into five queues. Five million events means five million copies in each of five queues. In Kafka there is one log and five offsets; adding a consumer does not add storage cost.

5. Per-entity ordering

The created, paid and shipped events for order 1042 are processed in order by a single queue with a single consumer. The moment you attach a second consumer to scale, those three messages can land on three different workers and shipped can be processed before paid.

In Kafka, key = orderId is enough: an order’s events go to the same partition, therefore the same consumer, in order.

6. The tooling around it

CDC with Debezium, writing to sinks with Kafka Connect, processing over the stream with Kafka Streams or Flink, schema evolution with a schema registry — this ecosystem is built around Kafka. If you are building an event backbone, sooner or later you need at least one of them.

Summary: the answer to “why Kafka”

None of the points above says “RabbitMQ cannot do this”; each of them has a solution. But notice that every one of those solutions is something you write and operate: a backfill script, a correction scan, a separate analytics store, the cost of duplicated queues, the consistent hash plugin.

In Kafka they fall out of the model itself — because the message is already sitting there, and its position is already on the consumer.

The reason to pick Kafka is not throughput. It is that the message is still valuable after it has been processed — that tomorrow a new consumer, a corrected piece of code or a new question will need the same history.

The “both have that” objection: one feature, two implementations

This is where the argument usually stalls. “RabbitMQ has fanout too”, “both have a dead letter queue”, “you can retry on either” — all three are true. At the level of a feature list, the difference disappears.

The difference is not in the list, it is in who owns the mechanism. RabbitMQ holds the delivery state, so the retry counter, the delay, dead-lettering and rejecting a message are broker features you configure. Kafka holds the position on the consumer, so those same things become application code and extra topics.

Here is what that looks like in three concrete places.

Fan-out: is it the message being copied, or the position?

RabbitMQ’s fanout exchange and Kafka’s consumer groups produce the same outcome: one event reaches five services. But they produce it differently:

RabbitMQ fanout:  the broker COPIES the message into 5 queues → 5 messages
Kafka groups:     one log, 5 groups each hold their own OFFSET → 1 message, 5 numbers

Every row below follows from that single difference:

So “RabbitMQ has fanout too” is true but incomplete: fanout distributes from now on. Kafka’s consumer groups do not distribute at all — they let everyone look at the same history at their own pace.

Retry and backoff: what is being blocked while you wait?

“Both can retry” is also true. But the real question about retry is: what is blocked for the duration of the wait?

In RabbitMQ

The simplest route is nack(requeue=true) — and there is a trap worth knowing here: a requeued message is put back not at the tail of the queue but as close to its original position as possible. Which means it is redelivered almost immediately. On a permanent failure that is a hot loop running thousands of attempts a second.

For real backoff you build a ladder of delay queues:

work-queue ──nack(requeue=false)──▶ work-dlx ──▶ retry-5s   (TTL=5s, no consumer)
                                                     │ TTL expires → dead-letter
                                                     ▼
                                                work-exchange ──▶ work-queue

You declare three queues — retry-5s, retry-30s, retry-5m — and pick which one to send to based on the attempt count in the x-death header. That counter is kept by the broker; you do not carry it:

List<Map<String, ?>> death = (List<Map<String, ?>>)
        message.getMessageProperties().getHeaders().get("x-death");
long attempts = death == null ? 0 : (Long) death.get(0).get("count");

Alternatively the rabbitmq_delayed_message_exchange plugin gives you an arbitrary per-message delay and removes the ladder entirely.

In Kafka

There are two options, and choosing is a direct trade-off:

// Spring Kafka — pauses the partition and retries in place
new DefaultErrorHandler(recoverer, new ExponentialBackOff(1_000L, 2.0));

@RetryableTopic(attempts = "4",
                backoff = @Backoff(delay = 1_000, multiplier = 2.0))
@KafkaListener(topics = "report-jobs")
public void handle(ReportJob job) { ... }

The trade-off as a table:

The empty ordering column on the RabbitMQ rows is not an oversight: a queue with several consumers has no ordering guarantee to begin with, so retry breaks nothing. In Kafka retry becomes a trade-off precisely because there is a guarantee there to lose.

That is the real difference: in RabbitMQ, a retry is the message’s own problem. In Kafka, a retry costs you either the partition’s progress or the ordering guarantee.

DLQ: a broker feature, or a topic you write to?

“Both have a dead letter queue” is the most misleading sentence in this argument. In RabbitMQ the DLX is a broker mechanism; in Kafka the DLT is just another topic — and the thing writing to it is your consumer.

The most practical row is the fourth. If you want to know how many times a message has been tried in Kafka, you put that in a header, you carry it across retry topics, and you write it to the DLT. Any link in that chain that forgets produces an infinite retry loop.

Bonus: how much gets reprocessed when a consumer crashes?

This is the rarely discussed side of the retry argument, and the implementation difference is very sharp here:

RabbitMQ:  consumer dies → the channel closes
           → only that consumer's unacked messages are requeued
           → granularity: a message

Kafka:     consumer dies → rebalance
           → the new owner starts from the last committed offset
           → granularity: everything since that commit

A Kafka consumer running with enable.auto.commit=true and a five-second commit interval crashing means reprocessing five seconds of traffic in full. The same event in RabbitMQ means requeueing however many messages the prefetch was holding.

Both are at-least-once; how many messages “at least once” actually amounts to is not the same.

Wrapping this part up

All three headings land on the same sentence:

Because RabbitMQ tracks delivery state, retry, delay, dead-lettering and rejection are broker settings. Because Kafka tracks position, those same behaviours are application decisions, extra topics and extra headers.

“Both have it” is true. But one is configuration and the other is code — and it is the code you will be operating.

Six questions for the decision

Having seen both scenarios, the choice reduces to six questions. The answers usually pile up on one side; if they do not, it is probably not a problem one broker is meant to solve.

The same thing as a decision tree:

graph TD
    A["What is this message saying?"] -->|"'This happened' — a fact"| C{"Is the history needed?<br/>replay, new consumers,<br/>rebuilding projections"}
    A -->|"'Do this' — a job"| B{"How many parties process it?"}
    B -->|"Several, independently"| C
    B -->|"A single worker"| D{"Per-message control?<br/>priority, delay,<br/>rejection, variable duration"}
    D -->|"Yes"| R["RabbitMQ"]
    D -->|"No, but the volume is very high"| K["Kafka"]
    C -->|"Yes"| K
    C -->|"No"| R

The general call

Reducing the six questions to concrete needs:

On where to start: if what you have is a classic task queue need, RabbitMQ is the more natural starting point. If you need event streaming, replay or high-volume stream processing, it makes more sense to evaluate Kafka.

The two are complementary more than competing; large systems use both — Kafka for the event backbone, RabbitMQ for the work queues.

Quick comparison

Wrapping up

Kafka and RabbitMQ approach the same problem with different models.

In Kafka messages stay in the log and consumers track their own position. That model is strong where you need replay, independent consumer groups and event streaming.

In RabbitMQ the broker tracks each message’s delivery state. Ack, requeue, routing and DLX give you a more natural model for task queue scenarios.

“Which one is faster?” turns out not to be a useful question. Once you build out concrete scenarios on both sides, the question that emerges is:

Is this message still valuable after it has been processed?

If it is — if tomorrow a new consumer, a corrected piece of code or a new question will need the same history — the message is a fact, and it belongs in a log: Kafka.

If it is not — if the work is finished when it is done, and what matters is that it happens once and eventually for certain, that it can be prioritised, deferred and rejected — the message is a job, and it belongs in a queue: RabbitMQ.

Using both in the same architecture is perfectly normal, too; most systems have both kinds of message in them anyway.