Kafka vs RabbitMQ vs SQS: Log or Queue, Not Speed

Somebody on your team is about to say “we should just use Kafka.” They will say it in a design review, they will point at a throughput number, and everyone will nod, because Kafka is what serious companies use. Six months later you own a cluster, a partition strategy, and a rebalancing bug, and your peak traffic is forty messages a second. I have sat in that meeting more times than I want to admit.

The throughput argument is almost never the real argument. All three of the usual candidates will move more messages than your product will ever generate. What actually separates them is a much smaller question, and it is a question about data structures: after a consumer reads a message, should the message still be there?

A log says yes. The message stays, each consumer tracks its own position, and you can rewind. A queue says no. The message is removed the moment a consumer acknowledges it, and it is gone. Everything else, ordering, routing, retries, pricing, falls out of that one answer.

A log keeps the message. A queue forgets it.

Kafka's own introduction is blunt about this. Events “can be read as often as needed” and, unlike traditional messaging systems, “are not deleted after consumption.” Instead you set a per-topic retention window, after which old events are discarded.^[1] The default is retention.msat 604,800,000 milliseconds, which is seven days, and the config reference describes that setting as “an SLA on how soon consumers must read their data.”^[3]

That works because Kafka does not track what has been consumed on the broker. A partition is consumed by exactly one consumer inside a group at a time, so the consumer's position is “just a single integer, the offset of the next message to consume.”^[2] Acknowledgment becomes a number you checkpoint instead of state the broker has to hold per message. And then the design doc drops the line I think about constantly: a consumer can rewind to an old offset and re-consume data, which “violates the common contract of a queue, but turns out to be an essential feature for many consumers.”^[2]

The first time I read that I laughed, because it is the whole argument stated by the people who built the thing. Kafka is not a better queue. It is deliberately not a queue.

Now the other side. In the AMQP 0-9-1 model that RabbitMQ implements, “a broker will only completely remove a message from a queue when it receives a notification for that message.”^[5] The acknowledgment is the delete. RabbitMQ's own reliability guide puts it in the same terms: a consumer ack confirms the delivery was processed “so the delivered message can be marked for future deletion.”^[6] SQS works the same way from the caller's side. You receive, you work, you call DeleteMessage, and the message stops existing.

The three systems on one axis

Here is where each one actually sits, with the numbers that matter rather than the ones on the landing page.

Apache Kafkais a partitioned append-only log. A topic is spread across partitions on different brokers, events with the same key land in the same partition, and Kafka guarantees that any consumer of a given topic-partition reads that partition's events “in exactly the same order as they were written.”^[1] Read that guarantee carefully, because it is narrower than people remember. There is no global ordering across a topic. If you need two events to be processed in order, they have to share a key, and if too many events share a key you have built a hot partition.

RabbitMQ is a broker that routes. Publishers do not write to queues, they write to exchanges, and exchanges distribute copies to queues using bindings. A direct exchange matches the routing key exactly, a fanout exchange copies to every bound queue and ignores the key, and a topic exchange matches the key against a pattern.^[5] That routing layer is the thing Kafka does not have, and it is genuinely useful when one event needs to reach three different consumers with three different filters.

Amazon SQSis a queue you do not run. Standard queues support “a nearly unlimited number of API calls per second” and give you at-least-once delivery, with the honest caveat printed right in the docs: “more than one copy of a message might be delivered, and messages may occasionally arrive out of order.”^[9] FIFO queues give you strict ordering, but only within a message group ID, and messages in different groups may be processed out of order relative to each other.^[10]

Consumer groups, and the rebalance nobody budgets for

Kafka's parallelism model is the part that bites teams after launch. All consumers sharing a group.id form one consumer group, and partitions are divided among them. If a process fails, its partitions are reassigned to other consumers in the group. If a new consumer joins, partitions are moved to it. Adding partitions to a subscribed topic triggers the same thing.^[4]

That reassignment is a rebalance, and because each partition is consumed by exactly one consumer in a group at a time, your partition count is a hard ceiling on how many workers can share the load.^[2] Twelve partitions means twelve useful consumers. The thirteenth pod sits there doing nothing while your dashboard shows thirteen healthy replicas. I have watched a team scale that deployment to twenty and wonder why the lag graph did not move.

Here is the current detail worth knowing. Kafka 4.0 made the next-generation rebalance protocol from KIP-848 generally available, and it is fully incremental, with no global synchronization barrier, which is exactly the stop-the-world pause older clusters are famous for.^[8] It is enabled on the server automatically. It is not enabled on the consumer by default, you have to set group.protocol to consumer.^[8] The fix shipped. Most clusters are still running without it because nobody read the release notes.

SQS has none of this, which is the actual selling point. There is no group membership, no assignment protocol, and no ceiling on consumer count for standard queues. You add workers and they pull. If you want per-key ordering you use a FIFO queue and a message group ID, and parallelism becomes a function of how many distinct groups you have.^[10]

The settings that decide whether this works at 3am

Every one of these systems is easy in the happy path. The differences show up when a consumer dies holding a message, and that is where the defaults matter.

SQS visibility timeout. When you receive a message it becomes invisible to other consumers for a fixed window: 30 seconds by default, 12 hours maximum.^[11] If your handler takes longer than the timeout and has not deleted the message, SQS makes it visible again and somebody else picks it up. A 90-second job on a default queue does not fail loudly. It succeeds three times.

Dead-letter queues. SQS moves a message to a DLQ after maxReceiveCount receives, set in the redrive policy.^[12] RabbitMQ dead-letters a message when it is rejected without requeue, when its TTL expires, when the queue exceeds a length limit, or when it exceeds the quorum queue delivery limit, which has defaulted to 20 since RabbitMQ 4.0 (before that it was unlimited).^[13] Kafka has no built-in dead-letter concept at all. You write that yourself.

Requeue order. RabbitMQ tries to put a requeued message back in its original position, and if it cannot, because other consumers acknowledged concurrently, the message goes closer to the head of the queue.^[6] So a queue you believed was ordered quietly is not, the first time a consumer nacks under load. That is worth pairing with a circuit breaker on the failing dependency, because a requeue loop and a dying downstream service make each other worse.

Duplicates. SQS FIFO deduplicates on a message deduplication ID, and the window is exactly 5 minutes.^[14] AWS calls this exactly-once processing, and inside that window it is. Outside it, or when your worker crashes after doing the work but before deleting the message, the message comes back. This is the same at-least-once world every broker lives in, covered in more depth in the delivery guarantees explainer, and the fix is not broker configuration. It is an idempotency key on your handler.

What you run versus what you rent

Now the part that decides more architectures than anyone admits in the design doc. Take a real service moving 10 million messages a month, which is roughly four a second. Each message costs three SQS API calls: send, receive, delete. That is 30 million requests. The first million every month are free, and standard requests are $0.40 per million in US East (N. Virginia), so the bill is $11.60.^[15]

The comparable managed Kafka is Amazon MSK. A minimum production cluster is three brokers, and kafka.m5.largeis $0.21 per broker hour in the same Region. Three brokers times 730 hours is $459.90 a month, before the $0.10 per GB-month of storage, and before anybody's salary.^[16]

And the dollar figure is the small half of the cost. The cluster is the other half: partition counts you have to guess before you have traffic, broker upgrades, disk forecasting against a seven-day retention window, and one engineer who becomes The Kafka Person and cannot take a vacation during peak season. None of that appears in the pricing page comparison people paste into the design doc.

This is the same trap as reaching for service boundaries too early, which I got into in the monolith versus microservices piece. The technology is not wrong. The timing is.

How I would actually pick

Three questions, in order. Stop at the first one that gives you a clear answer.

1. Does anything need to read the same message twice? A new consumer backfilling from history, a reprocess after a bug, an analytics pipeline reading the same events as your billing service. If yes, you want a log, and that means Kafka or a RabbitMQ Stream. If no, stop here, you want a queue, and you just eliminated the option with an operations bill.
2. Does one message need to reach several different consumers with different rules?That is routing, and RabbitMQ's exchanges do it natively with direct, fanout and topic matching.^[5] Doing the same thing on SQS means Amazon SNS in front of several queues, which is fine but is a second service.
3. Does order actually matter, and at what granularity? Order per customer, per account, per document is a message group ID on SQS FIFO or a partition key on Kafka. Global order across every message is a single message group or a single partition, which caps you at one consumer and you should be very sure you need it. Remember the FIFO ceilings: 300 TPS per API action, or 3,000 messages per second with 10-message batches, and high throughput mode is 70,000 TPS in three Regions, 18,000 TPS in US East (Ohio) and Europe (Frankfurt), and 2,400 in most of the rest.^[11]

My honest default for a product that has not proven it needs anything else: SQS standard, a dead-letter queue with maxReceiveCount set to something like 5, a visibility timeout longer than your slowest handler, and idempotent consumers. It costs eleven dollars and nobody has to know how partitions work. Move to a log when a real replay requirement shows up, not when somebody wants Kafka on their resume.

The pattern I keep seeing is teams buying Kafka for throughput they will never reach, then paying the operational bill for a replay feature they never use. If that describes your cluster, the honest move is not a benchmark. It is asking your team when anyone last reset an offset, and sitting with the answer.