How should I interpret these management UI charts "produced" by Spring Batch jobs?

[kvlttrve]

Community Support Policy

• I have read RabbitMQ's Community Support Policy

RabbitMQ version used

4.0.3

How is RabbitMQ deployed?

Community Docker image

Steps to reproduce the behavior in question

Hello guys, I am running a spring batch job with rabbitmq. I am writing statistics of my batch job to a table and have batch status as IN_PROGRESS, COMPLETED, etc. The issue I am having is my batch jobs get stuck in IN_PROGRESS on and off.

For example: in the attachment, for a blue spike (Deliver manual ack) in Message rate, if there no accompanying green spike (Consumer ack), then there will no red spike in Queued messages and the batch job will be stuck in IN_PROGRESS. However, for every blue spike that has accompanying green spike, it will have a red spike and the batch job will be COMPLETED.

What could be the issue? When I hit Get messages in the dashboard, it says the queue is empty.

Many Thanks.

[michaelklishin]

@kvlttrve the most likely issue is the overlap of two lines on the chart. Only one line will be visible. Management UI was never meant to be a best-in-class monitoring tool and the UI parts of it are 14 years old without significant visual changes.

If Spring AMQP acknowledges messages instantly and the batch is small enough, the number and rates of delivered and acknowledged metrics can be identical, which means they will be charted as two identical lines.

Take a traffic capture and inspect it with Wireshark, or try Prometheus, optionally with Grafana or query the HTTP API for relevant metrics directly.

Spring AMQP should expose some metrics via Micrometer.

[kvlttrve]

Thank you for your reply. Seeing activity in Message queue only when there is consecutive activity in message delivery and consumer ack and the subsequent completion of batch statistics made me post this here. However, like you mentioned my batch is small and messages are probably being acknowledged instantly.

However, if the issue lies in my application of rabbitmq then what could be some of the common reasons for my batch statistics being not written to table on and off? For example, if one batch is incomplete then subsequent batches will also be incomplete until one batch job completes successfully, then subsequent batches will also be successful until one becomes incomplete and the cycle continues. Also, the initial batches after I start my main application are always incomplete. If there was problem in my code, then I would expect either all of them to work or all of them to fail.

What do you think? How should I go about diagnosing this issues?

[michaelklishin]

Like I said, the problem is not with a lack of metrics, it's that they are identical and are charted in a way that only one can be seen.

I don't have any information about your apps and won't help you with that (no need to explain). I'd start with a traffic capture to see what is really being sent by both RabbitMQ and your application.

[michaelklishin]

Wow, I apparently have recommended the traffic capture twice now in this thread :D So yeah, that and the Prometheus endpoint to compare (you can hook up Grafana for visualization or just take a look at the data).

Not much else to add.