Nine years ago, my former boss Ajith Kumar asked me to run an impact analysis on replacing ActiveMQ with Kafka in our integration layer. We went ahead. Within a year we had replaced both ActiveMQ and IBM InfoSphere Streams across the stack.
I was thrilled. Duplicate messages for multiple consumers, gone. Back-pressure problems that used to eat entire afternoons, gone. Fan-out was fast, consumers were decoupled, and for a long time I genuinely believed Kafka had made traditional message queues obsolete. Broadcast, multiple consumers, retries. What else was there - I would smugly think.
A different work parttern - it turns out. It wasn't clear until I worked with a telco customer on their Kafka-based order processing platform that the distinction became crystal clear.
Two ways to deliver a message, and they are not interchangeable
Picture a party. Someone shouts "free pizza in the kitchen" and everyone who wants pizza goes and gets some. Nobody coordinates who takes which slice, nobody reports back, the shout just reaches whoever was listening. That is Kafka's publish-subscribe model. A producer writes to a topic, and every consumer group that subscribes reads the full stream independently, at its own pace, tracking its own offset. Ten downstream systems can all read the same event, none of them blocking the others, none of them aware the others exist.
Now picture a whiteboard with one line on it: "clean the kitchen." (the people at the party sober up). The first person free picks it up, does it, and wipes it off the board so nobody duplicates the work. If they get pulled away halfway through, the task goes back up for the next person. That is a queue. A message is a task. It goes to exactly one worker. Once that worker finishes it, the message is acknowledged and gone. If the worker fails partway through, the message goes back for someone else to pick up. Ownership is explicit at the level of a single message, not a partition or a consumer group.
The pizza shout tells you the message reached everyone who was listening. It says nothing about whether any one of them actually finished eating, let alone cleaned up after. The whiteboard tells you exactly which person holds the task right now, whether it is done, and what happens if it is not. You see the difference between workload patterns - one is broadcast while the other requires task coordination. You cannot get task-level ownership out of a broadcast model, no matter how cleverly you wire the consumer groups.
An epiphany in the Queue - the moment when I truly started appreciating the pub-sub and queue semantics.
Last year, I was helping a telco client build observability and monitoring layer around their Confluent Kafka setup. Out of curiosity, I asked how they used Kafka (in a professional and monitoring way 🤓).
Their order processing pipeline moved through four stages: validation, provisioning, billing, activation. Each stage had its own topic, and an order only moved forward if the current stage succeeded.
The problem showed up on failure. If validation failed for a transient reason, a database timeout during a credit check, the only way to retry was to republish the same message back onto the validation topic.
Me: "How long do you retry?" Them: "We increment a counter as a part of the Kafka message. If it crosses a threshold, we stop and send it to a dead-letter topic."
They tracked retry count in a custom header field. When it crossed their limit, they routed the message to a dead-letter topic by hand.
I asked how long they had been running it this way. Over a year, across roughly forty thousand orders a day. Nobody had questioned it because it worked albeit awkwardly. But every retry loop, every dead-letter routing decision, every piece of per-message state was something they had built themselves on top of Kafka. Basically, they were running a queue. Just that they had to write all the queue logic by hand.
That is the actual cost of forcing pub-sub to behave like a queue. Not that it fails. It works, but every team ends up reinventing the same retry and acknowledgment machinery, slightly differently, with slightly different bugs. The moment I saw that pattern clearly, I finally understood why the queue crowd had been saying "it depends" for years while I was busy being smug about Kafka. That was a good moment. It's the kind of thing you want to go tell someone about.
Why is KIP-932 so important
Apache Kafka's answer to all of this is KIP-932, known as Share Groups, and it reached general availability in Kafka 4.2 in February 2026. It took about two years from proposal to GA.
A share group lets multiple consumers process records from the same partition cooperatively, instead of the one-consumer-per-partition rule that ordinary consumer groups enforce. Each record gets an individual acquisition lock, thirty seconds by default, and the consumer that holds it can take one of four actions: acknowledge it as done, release it back for another attempt, reject it as unprocessable, or renew the lock if it needs more time. If none of those happen before the lock expires, the record becomes available to another consumer automatically. The broker tracks delivery attempts per record, five by default, and a message that exceeds that count is treated as unprocessable.
Here is the part that took me a moment to get straight: none of this is a topic setting. Topics do not get flagged as pub-sub topics or queue topics. The behavior lives entirely in how the consumer connects, through a single configuration value, group.type, set to either consumer or share. A share group is created with the exact same consumer API Kafka has always supported, just pointed at a different type. Which means the same topic can carry both patterns at once. Your existing consumer group can keep reading every message in order for analytics, completely untouched, while a share group of retry-handling workers reads from the same topic in parallel, acknowledging or releasing individual messages, and neither one knows the other exists. Adopting share groups is additive, not a migration. You are not converting a topic, you are pointing a new kind of consumer at it.
One more detail worth knowing if you go this route: a share group does not divide work by partition the way a consumer group does. It treats the whole topic as a single pool, every consumer in the group can pull from every partition, there is no sticky assignment. That is exactly what buys you the whiteboard behavior, any free worker can pick up any task, but it also means the ordering guarantee a partition normally gives you is gone the moment you attach a share group to it.
You see how this replaces the custom header counter and the manual republish-to-same-topic loop with something the broker does for you. The retry count lives in the broker's own tracking, not in application code three teams have to keep in sync. I would have loved to hand this feature to that client a year earlier.
The trade-off nobody should skip past
Ordinary Kafka consumer groups guarantee that messages within a partition are processed in the order they were written. That guarantee is what makes Kafka trustworthy for anything where sequence matters, an account balance update, a state transition that depends on the one before it.
Share groups do not give you that. Because multiple consumers can pull from the same partition at once, and because a released or rejected message can be redelivered out of turn, there is no guarantee that message five gets processed before message six. The KIP specifies ordering only within a single fetched batch, which is a narrow guarantee and one consumers do not control directly.
This is not a flaw in the implementation. It is the same trade-off traditional queues have always made. A queue optimizes for independent, parallelizable task completion. A log optimizes for sequence. You cannot have unordered parallel task pickup and strict sequential processing from the same mechanism at the same time. Share groups mean you no longer need a second messaging system to get the first property, but you still have to choose it deliberately, topic by topic, based on whether that topic's data actually depends on order.
Let us try to understand the trade-off better by comparing with RabbitMQ. A single consumer on a RabbitMQ queue gets strict FIFO. Nothing competes for messages, so nothing gets reordered. Add a second consumer and the broker still dequeues in FIFO, but delivery becomes round robin. A slow consumer can let a later message finish before an earlier one. RabbitMQ gives you two ways to fix this.
Share groups cannot give you order per key with parallelism today. A regular consumer group gets ordering because each partition sticks to one consumer, so keying your data by order ID keeps everything for that key in sequence. A share group treats the whole topic as one shared pool, so any free consumer can pick up any message from any partition, regardless of how you keyed it. If a workload genuinely needs order per key with parallelism, the answer is not a share group. It is a regular consumer group, partitioned by key, which is really Kafka's version of RabbitMQ's consistent hash exchange. This comes at the cost of queue semantics of course.
One gap still open: dead-letter queue support is not yet built in. KIP-1191, targeting Kafka 4.4 later in 2026, is meant to close it by automatically copying undeliverable records to a dedicated DLQ topic. Until then, teams adopting share groups are still writing that piece themselves, the same way my client did, just for a smaller slice of the problem.
Where this leaves the two-system question
Most enterprises I work with in the region run Kafka for streaming and a second system, RabbitMQ, Amazon SQS, IBM MQ, for task queues. That means two security models, two monitoring stacks, two on-call runbooks, and two places for governance to drift out of sync.
Share groups make it possible to collapse that into one platform for a meaningful set of workloads, specifically the ones where order does not matter and per-message retry does. Order processing intake, notification fan-out with individual delivery tracking, background job processing fed by an event stream. All good fits.
What share groups do not do is eliminate the need to think about ordering at design time. Infact, they make that decision more visible than it used to be, because now it is a configuration choice on the topic rather than an accident of which client library you happened to reach for. That is the right kind of forcing function.
Nine years ago I would have told you Kafka could do everything a queue could. That wasn't quite the right question, and I'm glad it's clear to me now. The question was never whether Kafka could deliver a message. It was whether it could tell you, for one specific message, exactly who owns it right now. That's a good place for Kafka to have landed.