Memory footprint of a RabbitMQ 3.8 node on Erlang 23.3 (or 24.x?) on CentOS 7

[NoraDing]

Community Support Policy

• I have read RabbitMQ's Community Support Policy
• I agree to provide all relevant information (versions, logs, rabbitmq-diagnostics output, detailed reproduction steps)

RabbitMQ version used

3.13.7 or older

Erlang version used

26.0.x

Operating system (distribution) used

CentOS 7.9.2009 x86_64

How is RabbitMQ deployed?

RPM package

rabbitmq-diagnostics status output

See https://www.rabbitmq.com/docs/cli to learn how to use rabbitmq-diagnostics

when the problem happened, I cannot get any result by running "rabbitmq-diagnostics status "

Logs from node 1 (with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

Logs from node 2 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

Logs from node 3 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

rabbitmq.conf

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find rabbitmq.conf file location

# PASTE rabbitmq.conf HERE, BETWEEN BACKTICKS

Steps to deploy RabbitMQ cluster

1. Install RabbitMQ & Erlang
   curl -s https://packagecloud.io/install/repositories/rabbitmq/rabbitmq-server/script.rpm.sh | sudo bash
   curl -s https://packagecloud.io/install/repositories/rabbitmq/erlang/script.rpm.sh | sudo bash
   yum -y update
   yum -y install epel-release
   yum remove erlang*
   yum remove erlang-erts-R16B-03.18.el7.x86_64
   yum -y install socat
   yum -y install rabbitmq-server-3.8.30*

2. Create/Modify configuration file
   cd /etc/rabbitmq
   cat > rabbitmq-env.conf
   NODE_PORT=5672
   export RABBITMQ_LOG_BASE=/rabbitmqlogs
   export RABBITMQ_MNESIA_BASE=/rabbitmqdata
   export RABBITMQ_USE_LONGNAME=true

eof

4. Configure log rotate
   cat > /etc/logrotate.d/rabbitmq-server
   /rabbitmqlogs/*.log {
   weekly
   missingok
   rotate 20
   size 200M
   compress
   notifempty
   }

5. Stop/disable Firewall
   systemctl stop firewalld.service
   systemctl disable firewalld.service

6. Start/enable RabbitMQ
   systemctl enable rabbitmq-server.service
   systemctl start rabbitmq-server.service
   systemctl status rabbitmq-server.service

RabbitMQ cluster setup

On Node 1:
7. Copy cookie file from master to all slaves, and make them identical

vim /var/lib/rabbitmq/.erlang.cookie #sample output RWKKKALCGQVVOZWCHJVO
cat /var/lib/rabbitmq/.erlang.cookie
chown rabbitmq:rabbitmq /var/lib/rabbitmq/*
chmod 400 /var/lib/rabbitmq/.erlang.cookie

8. Copy contents of erlang.cookie from Node 1 to NODE 2 ,3, 4 & 5
   (This is important to form an HA cluster with node 1)

9. Need to restart rabbitMQ on all node 2,3,4 & 5 servers after cookie change
   systemctl stop rabbitmq-server.service
   systemctl start rabbitmq-server.service

10. Join the cluster: on Node 2 ,3,4 & 5 by run the following

rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl join_cluster rabbit@
example: rabbitmqctl join_cluster [email protected]
rabbitmqctl start_app

13. Enable rabbitmq management plugin

on all 3 hosts, enable management plugin
rabbitmq-plugins enable rabbitmq_management

14. Enable other important Plugins
    rabbitmq-plugins enable rabbitmq_top
    rabbitmq-plugins enable rabbitmq_federation
    rabbitmq-plugins enable rabbitmq_federation_management
    rabbitmq-plugins enable rabbitmq_shovel
    rabbitmq-plugins enable rabbitmq_shovel_management
    rabbitmq-plugins enable rabbitmq_prometheus

15. Check Plugins on all hosts
    rabbitmq-plugins list
    rabbitmq-plugins list --enabled

Steps to reproduce the behavior in question

We cannot reproduce this problem

advanced.config

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find advanced.config file location

# PASTE advanced.config HERE, BETWEEN BACKTICKS

Application code

# PASTE CODE HERE, BETWEEN BACKTICKS

Kubernetes deployment file

# Relevant parts of K8S deployment that demonstrate how RabbitMQ is deployed
# PASTE YAML HERE, BETWEEN BACKTICKS

What problem are you trying to solve?

We got alert about the rabbitmq service is down .
And then we try to login to the console, but it's hard to login in . And we cannot find any error logs. But we only can see the memory usage is high in one node,the other nodes' memory usage is normal.
We cannot find any way to solve this situation. So we restart the problem node and the cluster recovered

[mkuratczyk]

RabbitMQ 3.8 has been out of support for years.

[michaelklishin]

I have edited out some company-identifying information from the instructions.

@NoraDing we cannot know what consumes memory or other resources on your node. Start with Reasoning About Memory Footprint and Monitoring with Prometheus and Grafana that demonstrates that there are tools that monitor more than the total footprint (but rather specific sub-categories).

RabbitMQ 3.8 is five minors and close to 100 patches behind the latest 3.x release (3.13.7), which has been out of community support for over two months now. Given the gap in versions, I'd start with the Blue-Green Deployment guide because that's going to be the most practical upgrade option for you.

[michaelklishin]

@NoraDing and then there is CentOS 7, another piece of this puzzle that is completely out of support.

FWIW, our team still quietly produces CentOS 7 versions of zero dependency RPMs, so running RabbitMQ 4.0.x on Erlang 26.x on CentOS 7 is still possible as a worst case transitionary scenario.

[michaelklishin]

@NoraDing finally, you consume packages from PackageCloud, which our team no longer uses or updates. We don't know when PackageCloud might disactivate our account. Our current RPM repos do not provide 3.8.x but GitHub releases of both RabbitMQ server and our zero dependency Erlang RPM go back a good decade.

According to an older version of the docs, RabbitMQ 3.8 requires Erlang/OTP 23.

Then 23.3.x or 24.3.x releases should be an option for direct downloads instead of using PackageCloud. Not all of them have a CentOS 7 build but at least a few should.