One cluster node uses more memory than other peers

[NoraDing]

Community Support Policy

• I have read RabbitMQ's Community Support Policy
• I agree to provide all relevant information (versions, logs, rabbitmq-diagnostics output, detailed reproduction steps)

RabbitMQ version used

3.13.7 or older

Erlang version used

26.0.x

Operating system (distribution) used

CentOS 7.9.2009 x86_64

How is RabbitMQ deployed?

RPM package

rabbitmq-diagnostics status output

See https://www.rabbitmq.com/docs/cli to learn how to use rabbitmq-diagnostics

Status of node rabbit@hostname ...
Runtime

OS PID: 32337
OS: Linux
Uptime (seconds): 79820980
Is under maintenance?: false
RabbitMQ version: 3.8.30
Node name: rabbit@hostname
Erlang configuration: Erlang/OTP 23 [erts-11.2.2.10] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]
Erlang processes: 2804 used, 1048576 limit
Scheduler run queue: 2
Cluster heartbeat timeout (net_ticktime): 60

Plugins

Enabled plugin file: /etc/rabbitmq/enabled_plugins
Enabled plugins:

 * rabbitmq_delayed_message_exchange
 * rabbitmq_prometheus
 * rabbitmq_federation_management
 * rabbitmq_shovel_management
 * amqp10_client
 * amqp10_common
 * rabbitmq_federation
 * rabbitmq_top
 * prometheus
 * rabbitmq_management
 * amqp_client
 * rabbitmq_web_dispatch
 * cowboy
 * cowlib
 * rabbitmq_management_agent

Data directory

Node data directory: /rabbitmqdata/rabbit@hostname
Raft data directory: /rabbitmqdata/rabbit@hostname/quorum/rabbit@hostname
Config files


Log file(s)

 * /rabbitmqlogs/[email protected]
 * /rabbitmqlogs/rabbit@hostname_upgrade.log

Alarms

(none)

Memory

Total memory used: 2.1769 gb
Calculation strategy: rss
Memory high watermark setting: 0.4 of available memory, computed to: 13.4267 gb

allocated_unused: 1.6894 gb (74.76 %)
binary: 0.1275 gb (5.64 %)
plugins: 0.1113 gb (4.93 %)
mgmt_db: 0.0872 gb (3.86 %)
other_proc: 0.0684 gb (3.03 %)
queue_procs: 0.0682 gb (3.02 %)
code: 0.0296 gb (1.31 %)
metrics: 0.0218 gb (0.97 %)
other_system: 0.0186 gb (0.82 %)
connection_other: 0.014 gb (0.62 %)
other_ets: 0.0087 gb (0.38 %)
connection_readers: 0.0059 gb (0.26 %)
queue_slave_procs: 0.0027 gb (0.12 %)
msg_index: 0.0024 gb (0.1 %)
atom: 0.0015 gb (0.06 %)
mnesia: 0.0013 gb (0.06 %)
connection_channels: 0.0009 gb (0.04 %)
connection_writers: 0.0003 gb (0.01 %)
quorum_ets: 0.0 gb (0.0 %)
quorum_queue_procs: 0.0 gb (0.0 %)
reserved_unallocated: 0.0 gb (0.0 %)

File Descriptors

Total: 261, limit: 65439
Sockets: 199, limit: 58893

Free Disk Space

Low free disk space watermark: 0.05 gb
Free disk space: 407.2507 gb

Totals

Connection count: 199
Queue count: 103
Virtual host count: 11

Listeners

Interface: [::], port: 15672, protocol: http, purpose: HTTP API
Interface: [::], port: 15692, protocol: http/prometheus, purpose: Prometheus exporter API over HTTP
Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI to

Logs from node 1 (with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

Logs from node 2 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

Logs from node 3 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

rabbitmq.conf

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find rabbitmq.conf file location

# PASTE rabbitmq.conf HERE, BETWEEN BACKTICKS

Steps to deploy RabbitMQ cluster

1. Install RabbitMQ & Erlang
   curl -s https://packagecloud.io/install/repositories/rabbitmq/rabbitmq-server/script.rpm.sh | sudo bash
   curl -s https://packagecloud.io/install/repositories/rabbitmq/erlang/script.rpm.sh | sudo bash
   yum -y update
   yum -y install epel-release
   yum remove erlang*
   yum remove erlang-erts-R16B-03.18.el7.x86_64
   yum -y install socat
   yum -y install rabbitmq-server-3.8.30*

2. Create/Modify configuration file
   cd /etc/rabbitmq
   cat > rabbitmq-env.conf
   NODE_PORT=5672
   export RABBITMQ_LOG_BASE=/rabbitmqlogs
   export RABBITMQ_MNESIA_BASE=/rabbitmqdata
   export RABBITMQ_USE_LONGNAME=true

eof

4. Configure log rotate
   cat > /etc/logrotate.d/rabbitmq-server
   /rabbitmqlogs/*.log {
   weekly
   missingok
   rotate 20
   size 200M
   compress
   notifempty
   }

5. Stop/disable Firewall
   systemctl stop firewalld.service
   systemctl disable firewalld.service

6. Start/enable RabbitMQ
   systemctl enable rabbitmq-server.service
   systemctl start rabbitmq-server.service
   systemctl status rabbitmq-server.service

RabbitMQ cluster setup

On Node 1:
7. Copy cookie file from master to all slaves, and make them identical

vim /var/lib/rabbitmq/.erlang.cookie #sample output RWKKKALCGQVVOZWCHJVO
cat /var/lib/rabbitmq/.erlang.cookie
chown rabbitmq:rabbitmq /var/lib/rabbitmq/*
chmod 400 /var/lib/rabbitmq/.erlang.cookie

8. Copy contents of erlang.cookie from Node 1 to NODE 2 ,3, 4 & 5
   (This is important to form an HA cluster with node 1)

9. Need to restart rabbitMQ on all node 2,3,4 & 5 servers after cookie change
   systemctl stop rabbitmq-server.service
   systemctl start rabbitmq-server.service

10. Join the cluster: on Node 2 ,3,4 & 5 by run the following

rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl join_cluster rabbit@
example: rabbitmqctl join_cluster [email protected]
rabbitmqctl start_app

13. Enable rabbitmq management plugin

on all 3 hosts, enable management plugin
rabbitmq-plugins enable rabbitmq_management

14. Enable other important Plugins
    rabbitmq-plugins enable rabbitmq_top
    rabbitmq-plugins enable rabbitmq_federation
    rabbitmq-plugins enable rabbitmq_federation_management
    rabbitmq-plugins enable rabbitmq_shovel
    rabbitmq-plugins enable rabbitmq_shovel_management
    rabbitmq-plugins enable rabbitmq_prometheus

15. Check Plugins on all hosts
    rabbitmq-plugins list
    rabbitmq-plugins list --enabled

Steps to reproduce the behavior in question

cannot reproduce this problem

advanced.config

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find advanced.config file location

# PASTE advanced.config HERE, BETWEEN BACKTICKS

Application code

# PASTE CODE HERE, BETWEEN BACKTICKS

Kubernetes deployment file

# Relevant parts of K8S deployment that demonstrate how RabbitMQ is deployed
# PASTE YAML HERE, BETWEEN BACKTICKS

What problem are you trying to solve?

The memory usage of one rabbitmq is far higher than other nodes from one same cluster .
Can you help confirm why the value " allocated_unused: 1.6894 gb (74.76 %) " is high?

With the days or hours going, the memory will be suddenly decrease and the node will be stuck. And we can solve it by restarting the node only.

here are some evidence for your reference.
The consumers is much more than other nodes and the number is still increasing.

[lukebakken]

The consumers is much more than other nodes and the number is still increasing.

Sounds like an application bug.

RabbitMQ 3.13.x is ineligible for free community support. Support for RabbitMQ 3.13.x versions requires a contract with Broadcom (https://github.com/rabbitmq/rabbitmq-server/blob/main/COMMUNITY_SUPPORT.md#who-is-eligible-for-community-support).

You must use the latest version of RabbitMQ and Erlang to be eligible for community support from Team RabbitMQ.

[HowardTwine]

@NoraDing I'd be very happy to learn more about your RabbitMQ usage and share more about how the Broadcom team could support your use of RabbitMQ in the future.

[michaelklishin]

This is a continuation to #12761, where the version used (3.8.30) is completely out of any kind of support (and has been for years) yet we have provided some guidance.

allocated_unused is an area reserve by the runtime for further. https://www.rabbitmq.com/docs/runtime#allocators explains how to tweak memory allocator flags. We do not know anything about the environment so we cannot possibly suggest why the default allocators decide to allocate that amount.

@NoraDing please migrate to RabbitMQ 4.0.x, the only series eligible for community support, and share relevant details in the discussion template.

Logs and metrics are the most important parts. If you use RabbitMQ for free and expect help with technical operations from the core team, please use a supported version and provide relevant details. Or buy a support subscription, but even commercial customers are asked to provide relevant information.

One screenshot is absolutely not enough. How you set up the Erlang cookie in a script is not particularly relevant for the question at hand, that part can be left out.

[michaelklishin]

Here is a similar discussion #12668 where one cluster node used more CPU resources than other nodes.

It explains what kind of questions the core team asks, what are the most typical scenarios, what metrics and documentation parts are relevant (well, for that question anyway).

This is about as much as we can suggest with a single screenshot, one rabbitmq-diagnostics memory_breakdown value (out of a about dozen) and an Erlang version.