Some messages in QQs are shows as pending consumer acknowledgements after consumer connection is closed

[saivikas-d]

Describe the bug

Am load testing testing 4.1 RMQ Cluster with Quorum Queue before we migrate from our existing 3.12 version.

We employ Blue Green deployment and before we cut-over to the new 4.1 cluster we wanted to load test the cluster in our environment and measure the throughput of our cluster.

RMQ Cluster details
Version - 4.1.0
Erlang Version. - 27.3.4

The load tests are performed using perf-test tool for Quorum Queues with replication factor of 3

While load testing am encountering the issue, when the load test process completes it run before sending a ACK to consumed messages. The Non ACK'ed messages remain in Unacked state for a very long time more than 40-45 minutes, and as long as these messages are in UnAcked state am unable to publish any new messages to the Queue. Effectively rendering the Queue inaccessible for the duration.

Am consistently able to reproduce this behavior.

Reproduction steps

1.Run the perf test using the following flags, replace username/password and cluster details

java -jar perf-test.jar --uri amqp://username:password@clusterdomain:5672 --size 1000 --confirm 1 --quorum-queue --time 300 --shutdown-timeout 20 --queue-pattern 'perf-test-%d' --queue-pattern-from 1 --queue-pattern-to 20 --producers 20 --consumers 20 --nio-threads 10 --producer-channel-count 1024 --consumer-channel-count 1024 --output-file test_result11.out

When the load test run completes you can observe from the cluster that the messages are in Unacked state

Rerun the perf test jar again when the messages are in Unacked state. You can observe that publish and consumer rates are 0

Sample output from Perf Test

test-110635-144, time 252.001 s, sent: 0 msg/s, confirmed: 0 msg/s, nacked: 0 msg/s, received: 0.50 msg/s, min/median/75th/95th/99th consumer latency: 737265067/737265067/737265067/737265067/737265067 µs, confirm latency: 0/0/0/0/0 µs

Even though the there are no active connections Quorum Queues with UnAcked messages shows consumer count in Queue page

Cluster Connections

Queue with Unacked messages

Expected behavior

Messages should go back to ready state immediately when Consumer connections die and should not block publishers from publishing new messages

Additional context

No response

[mkuratczyk]

Most likely this is not a bug, but an overloaded RabbitMQ - even though the clients disconnect, it still has a backlog of things to do.

Is this in any way a realistic workload for you? It's not necessarily the source of the problem, but:

• 1024 channels per connection is an overkill (you rarely need more than a few)
• 1024 consumers on a single queue is way above what most people need
• 1024 publishers per queues is quite a lot (although not unrealistic for certain scenarios)

I'm happy to take a look at potential performance issues/areas we could improve, but let's make sure we spend the time on things that matter, not benchmarks that are not representative of real workloads.

[mkuratczyk]

I ran it in my env and I don't see this behaviour, but performance issues are notoriously hard to reproduce. The behaviour heavily depends on the exact infrastructure you have, disk speeds, network speeds and sometimes even the exact timing of certain events.

Is the performance better using an older RabbitMQ version?

A few things I can suggest are:

• fewer consumers
• lower prefetch (while a higher value generally increases the throughput of a given consumers, having lots of consumers with a high prefetch each, in this case leads to RabbitMQ trying to serve 100k+ messages at once, even though it probably doesn't make anything faster)
• faster disks
• check how queues/leaders are balanaced
• capture flamegraphs (https://www.rabbitmq.com/blog/2022/05/31/flame-graphs)

[michaelklishin]

@saivikas-d I have explained what the rabbitmq-diagnostics observer data means.

As mentioned earlier, some of the values in this workload do not make much practical sense. There is no reason to use 1024 channels per publisher or consumers. Acknowledging one delivery at a time is a guaranteed way to slow down your consumer and keep the backlog of operations higher.

Application shutdown timeout is a reasonable thing to use but when a node is being pushed close to its limits, a low limit is guaranteed to cause some pending confirms (publisher confirms, consumer acknowledgements) on some connections and it can take a while for the related metrics to be cleaned up.

[saivikas-d]

@mkuratczyk

I ran similar test in 3.12.2 RMQ version using the same perf-test params and I don't see the issue there.

The performance is definitely better in 4.1 where we are able to achieve 70K publishes/sec and 70K consumes/sec where as in the 3.12 both publish and consume rates are around 30K

The RMQ cluster runs on Azure VMs Standard_F16s_v2 with Premium SSD data drives for 3.x and 4.x cluster setup

Queues are equally balanced as we use queue_leader_locator = balanced

I understand the consumer count is high per connection, we have experienced consumer slowness sometimes causing few thousand messages to go to unack state in production setup and we sometimes restart applications to force the messages to move back to ready state.

Rabbit 4.1 is definitely faster in producing and consuming messages. Based on tests the Quorum queue is holding on the consumers as active even when the underlying connection is closed. over a period of 40-45 mins it's slowly closing the consumers and the unacked messages are coming to ready state.

I now ran a test again with consumer prefetch count of 10 and messages are not in unacked state as expected.

Just for confirming when you ran the test against your cluster did it have any QOS setting in advanced.conf file

[mkuratczyk]

No, I tested with unlimited prefetch just like you originally did. But as I said - there's lots of factors that may impact the exact behaviour. Given that the throughput overall is much higher, I'd say you should be fine: in a real system, you don't publish as much as you can - you publish as much as you need to. And RabbitMQ 4.1 should be able to handle a higher throughput.

As for the behaviour once RabbitMQ is overloaded, I just realized we fixed one thing in 4.1.1 that could affect your case, so please try with 4.1.1 if you haven't yet.

Lastly, if you still believe there's something to improve, since we can't reproduce this, the next step is for you to capture flamegraphs of what the system is busy with.

[saivikas-d]

Thank you for your inputs @mkuratczyk, I will share the results once testing is done in 4.1.1

[michaelklishin]

The node without unacked messages is spending most of its time starting queue replicas, connections and channels (rabbit_limiter:init/1, rabbit_writer:enter_mainloop/1), while on the other two nodes most time is spent running PerfTest connections with certain names.

In other words, the only node not exhibiting this behavior is a node that does not serve client traffic (not yet or because PerfTest was pointed to its peers).