rabbitmq-streams list_stream_consumers fails

[ankitgr8]

Describe the bug

ENV --- 3 node rabbitmq cluster running 3.13.7 version... in a kuberneetes environment (AKS environement - Azure Kuberneetes Service). while generating the rabbitmq docker file we enable the stream plugin in offline node
COMMAND - rabbitmq-plugins enable --offline rabbitmq_stream

Once the cluster is UP.. the below command fails.

[root@rabbitmq-cluster-0 rabbitmq]# rabbitmq-streams list_stream_consumers
Listing stream consumers ...
%Protocol.UndefinedError{protocol: Enumerable, value: {:badrpc, {:EXIT, {:normal, {:gen_statem, :call, [#PID<21085.212228472.0>, {:info
, [:vhost]}, :infinity]}}}}, description: ""}

But few other commands are still working
rabbitmq-streams [--node ] [--longnames] [--quiet] list_stream_tracking [--all | --offset | --writer] [--vhost ]
[root@rabbitmq-cluster-0 rabbitmq]# rabbitmq-streams list_stream_tracking gms2ita
Listing tracking information for stream gms2ita ...
┌────────┬────────────────────┬────────────────┐
│ type │ name │ tracking_value │
├────────┼────────────────────┼────────────────┤
│ offset │ dcs-kafka-consumer │ 69854 │
└────────┴────────────────────┴────────────────┘

rabbitmq-plugin DETAILS

[root@rabbitmq-cluster-0 rabbitmq]# rabbitmq-plugins list | grep stream
[E*] rabbitmq_stream 3.13.7
[ ] rabbitmq_stream_management 3.13.7
[root@rabbitmq-cluster-0 rabbitmq]#

Why few command from the same plugin work and some not?

Reproduction steps

1. 3 node rabbitmq cluster k8s
2. enable the stream plugin in offline mode while generating the docker file
3. Once the cluster is up using the same docker file-- the rabbitmq cluster was upgraded .. not a fresh installation and the upgrade was rolling upgrade
4. The cli rabbitmq-stream list_stream_consumer does not work
   ...

Expected behavior

rabbitmq-streams list_stream_consumers fails should work post rabbitmq-stream plugin is enabled

Additional context

No response

[ankitgr8]

Do we want any specific PORT for rabbitmq-stream cli to be open for talking to the nodes accross cluster nodes

[acogoluegnes]

Do we want any specific PORT for rabbitmq-stream cli to be open for talking to the nodes accross cluster nodes

This is likely to be the problem. list_stream_consumers contacts all the cluster nodes, whereas list_stream_tracking contacts only the local node.

[ankitgr8]

rabbitmqctl list_queues is working fine.. this cli also works accross nodes.. and is working fine.. and the same port which rabbitmqctl list_queues is using i think the same port is used in rabbit-stream also...

My suspect is that during rolling upgrade of rabbitmq we enable the rabbit-stream plugin... is causing some issue

[michaelklishin]

@ankitgr8 you haven't provided any evidence of a bug.

list_* commands connect to all nodes in the cluster and combined the aggregated result. Some list_* command may choose to contact only one node but that's a minority of them.

:badrpc for exception is a pretty direct clue that suggests that CLI tools could not contact one of the nodes. We cannot know why, see https://www.rabbitmq.com/docs/troubleshooting-networking and https://www.rabbitmq.com/docs/networking#ports. The shared secret is unlikely to be the culprit simply because it cannot be different between N cluster nodes.

RabbitMQ 3.13 is out of community support so please find an executable way to reproduce on 4.0.x.

[ankitgr8]

Thanks for the pointers to doc. but can you share the doc link or steps to enable logs for CLI error

Although my env says, every required port is open and rabbitmq list_queues is working fine

[root@rabbitmq-cluster-0 rabbitmq]# rabbitmqctl list_queues | grep gms2
gms2ita214798
[root@rabbitmq-cluster-0 rabbitmq]#

WHAT is failing is
[root@rabbitmq-cluster-0 rabbitmq]# rabbitmqctl list_stream_consumers
Listing stream consumers ...
%Protocol.UndefinedError{protocol: Enumerable, value: {:badrpc, {:EXIT, {:normal, {:gen_statem, :call, [#PID<21085.149669084.4>, {:info
, [:vhost]}, :infinity]}}}}, description: ""}

NODE 0
[root@rabbitmq-cluster-0 rabbitmq]# rabbitmq-diagnostics listeners
Asking node [email protected] to report its protocol listeners ...
Interface: [::], port: 5551, protocol: stream/ssl, purpose: stream/ssl
Interface: [::], port: 15692, protocol: http/prometheus, purpose: Prometheus exporter API over HTTP
Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Interface: [::], port: 5671, protocol: amqp/ssl, purpose: AMQP 0-9-1 and AMQP 1.0 over TLS
[root@rabbitmq-cluster-0 rabbitmq]#

NODE 1

[root@rabbitmq-cluster-1 rabbitmq]# rabbitmq-diagnostics listeners
Asking node [email protected] to report its protocol listeners ...
Interface: [::], port: 5551, protocol: stream/ssl, purpose: stream/ssl
Interface: [::], port: 15692, protocol: http/prometheus, purpose: Prometheus exporter API over HTTP
Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Interface: [::], port: 5671, protocol: amqp/ssl, purpose: AMQP 0-9-1 and AMQP 1.0 over TLS

NODE 2

[root@rabbitmq-cluster-2 rabbitmq]# rabbitmq-diagnostics listeners
Asking node [email protected] to report its protocol listeners ...
Interface: [::], port: 5551, protocol: stream/ssl, purpose: stream/ssl
Interface: [::], port: 15692, protocol: http/prometheus, purpose: Prometheus exporter API over HTTP
Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Interface: [::], port: 5671, protocol: amqp/ssl, purpose: AMQP 0-9-1 and AMQP 1.0 over TLS

[michaelklishin]

There is --print-stacktrace but badrpc strongly suggest a connectivity issue. Perhaps some ports are not open on some of the target nodes.

Anyhow, we will not troubleshoot networking for you and "3.13.x is out of community support" means just that, it is out of support. You have received much more advice than what you are entitled to unless you have a support license.