Maximum number of Erlang processes and atoms can be configured via environment variables #1528

gerhard · 2018-02-23T18:06:24Z

Proposed Changes

Allow maximum number of Erlang processes to be configured via an environment variable
Allow maximum number of Erlang atoms to be configured via an environment variable

Types of Changes

Bugfix (non-breaking change which fixes issue #NNNN)
New feature (non-breaking change which adds functionality) - partially relevant for Unreasonably high lock contention when a node restarts and there are a lot of [exclusive] queues delete #1513
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation (correction or otherwise)
Cosmetics (whitespace, appearance)

Checklist

I have read the CONTRIBUTING.md document
I have signed the CA (see https://cla.pivotal.io/sign/rabbitmq)
All tests pass locally with my changes
I have added tests that prove my fix is effective or that my feature works
I have added necessary documentation (if appropriate)
Any dependent changes have been merged and published in related repositories

This property is essential for nodes that need to handle many connections / channels / queues. Related to #1513

This was the only property in SERVER_ERL_ARGS that was not configurable. Even though it's unlikely that this will need to be increased above the existing 5 million default, we are consistent in allowing all properties in SERVER_ERL_ARGS to be cofigured.

gerhard · 2018-02-23T18:08:02Z

Proposing to add this to v3.6.x & v3.8.x

michaelklishin · 2018-02-23T18:26:41Z

It can already be done using RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS but I have no objections to these two.

lukebakken · 2018-02-23T18:33:56Z

I'd say apply this to v3.7.x and master since RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS provides this function for v3.6.x.

michaelklishin · 2018-02-23T18:36:28Z

If this is driven by the needs of the BOSH release (which is a guess, perhaps @gerhard can elaborate), I have no objections to having this in 3.6.x as well.

gerhard · 2018-02-23T19:07:43Z

Yes, I was actually thinking about RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS and thought that it was weird to have some erl properties such as +zddbl & +stbt configurable explicitly, and others such as +P & +t only configurable as overrides via $RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS.

gerhard · 2018-02-23T19:10:23Z

The BOSH release supports both RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS & specific properties, so it's not an issue - both approaches will yield the same result. This PR is driven by making all properties in SERVER_ERL_ARGS equally configurable.

This will not work in all RabbitMQ versions, use rabbitmq-server.additional_erl_args for greater cross-version compatibility. Related to rabbitmq/rabbitmq-server#1528

michaelklishin

When I try to run a node on this branch, it fails with

gmake run-broker

 GEN    /var/folders/gp/53t98z011678vk9rkcb_s6ph0000gn/T//rabbitmq-test-instances/rabbit/enabled_plugins
All plugins have been disabled.
Applying plugin configuration to rabbit@warp10...
Plugin configuration unchanged.
 GEN    /var/folders/gp/53t98z011678vk9rkcb_s6ph0000gn/T//rabbitmq-test-instances/test.config
MAKE="gmake" ERL_LIBS="/Users/antares/Development/RabbitMQ/umbrella.git/deps/rabbit/apps:/usr/local/Cellar/elixir/1.6.0_1/bin/../lib/elixir" RABBITMQ_NODENAME="rabbit" RABBITMQ_NODE_IP_ADDRESS="" RABBITMQ_NODE_PORT="" RABBITMQ_PID_FILE="/var/folders/gp/53t98z011678vk9rkcb_s6ph0000gn/T//rabbitmq-test-instances/rabbit/rabbit.pid" RABBITMQ_LOG_BASE="/var/folders/gp/53t98z011678vk9rkcb_s6ph0000gn/T//rabbitmq-test-instances/rabbit/log" RABBITMQ_MNESIA_BASE="/var/folders/gp/53t98z011678vk9rkcb_s6ph0000gn/T//rabbitmq-test-instances/rabbit/mnesia" RABBITMQ_MNESIA_DIR="/var/folders/gp/53t98z011678vk9rkcb_s6ph0000gn/T//rabbitmq-test-instances/rabbit/mnesia/rabbit" RABBITMQ_SCHEMA_DIR="/var/folders/gp/53t98z011678vk9rkcb_s6ph0000gn/T//rabbitmq-test-instances/rabbit/schema" RABBITMQ_GENERATED_CONFIG_DIR="/var/folders/gp/53t98z011678vk9rkcb_s6ph0000gn/T//rabbitmq-test-instances/rabbit/config" RABBITMQ_PLUGINS_DIR="/Users/antares/Development/RabbitMQ/umbrella.git/deps/rabbit/plugins" RABBITMQ_PLUGINS_EXPAND_DIR="/var/folders/gp/53t98z011678vk9rkcb_s6ph0000gn/T//rabbitmq-test-instances/rabbit/plugins" RABBITMQ_SERVER_START_ARGS="" RABBITMQ_ENABLED_PLUGINS_FILE="/var/folders/gp/53t98z011678vk9rkcb_s6ph0000gn/T//rabbitmq-test-instances/rabbit/enabled_plugins" \
  RABBITMQ_ALLOW_INPUT=true \
  RABBITMQ_CONFIG_FILE=/var/folders/gp/53t98z011678vk9rkcb_s6ph0000gn/T//rabbitmq-test-instances/test \
  /Users/antares/Development/RabbitMQ/umbrella.git/deps/rabbit/scripts/rabbitmq-server
bad number of processes =1048576
Usage: beam.smp [flags] [ -- [init_args] ]
The flags are:

michaelklishin · 2018-02-23T19:28:54Z

scripts/rabbitmq-env

@@ -87,8 +87,14 @@ DEFAULT_SCHEDULER_BIND_TYPE="db"
 DEFAULT_DISTRIBUTION_BUFFER_SIZE=128000
 [ "x" = "x$RABBITMQ_DISTRIBUTION_BUFFER_SIZE" ] && RABBITMQ_DISTRIBUTION_BUFFER_SIZE=${DEFAULT_DISTRIBUTION_BUFFER_SIZE}

+DEFAULT_MAX_NUMBER_OF_PROCESSES=1048576
+[ "x" = "x$RABBITMQ_MAX_NUMBER_OF_PROCESSES" ] && RABBITMQ_MAX_NUMBER_OF_PROCESSES=="${DEFAULT_MAX_NUMBER_OF_PROCESSES}"


== seems to be bash-specific and is not available in zsh.

My bad, definitely a typo.

… with RABBITMQ_, adjust Unix and Windows files accordingly

I missed some errors

lukebakken · 2018-02-23T19:57:42Z

OK I tested my latest changes by creating a rabbitmq-env.conf file with these contents:

lbakken@shostakovich ~/issues/rabbitmq/rabbitmq-server/gh-1528
$ cat rabbitmq-env.conf 
SCHEDULER_BIND_TYPE=u
DISTRIBUTION_BUFFER_SIZE=65536
MAX_NUMBER_OF_PROCESSES=131072
MAX_NUMBER_OF_ATOMS=196608

Then running from this branch with this command:

make RABBITMQ_NODENAME='rabbit@shostakovich' RABBITMQ_CONF_ENV_FILE=$HOME/issues/rabbitmq/rabbitmq-server/gh-1528/rabbitmq-env.conf run-broker

ps -ef output shows that the settings were applied:

$ ps -ef|fgrep beam.smp
lbakken  25122 22593 15 11:55 pts/2    00:00:02 /home/lbakken/development/erlang/installs/19.3/erts-8.3/bin/beam.smp -W w -A 128 -P 131072 -t 196608 -stbt u -zdbbl 65536 -K true -- -root /home/lbakken/development/erlang/installs/19.3 -progname erl -- -home /home/lbakken -- -kernel shell_history enabled -pa /home/lbakken/development/rabbitmq/rabbitmq-server/ebin -s rabbit boot -sname rabbit@shostakovich -boot start_sasl -config /tmp/rabbitmq-test-instances/test -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit lager_log_root "/tmp/rabbitmq-test-instances/rabbit@shostakovich/log" -rabbit lager_default_file "/tmp/rabbitmq-test-instances/rabbit@shostakovich/log/[email protected]" -rabbit lager_upgrade_file "/tmp/rabbitmq-test-instances/rabbit@shostakovich/log/rabbit@shostakovich_upgrade.log" -rabbit enabled_plugins_file "/tmp/rabbitmq-test-instances/rabbit@shostakovich/enabled_plugins" -rabbit plugins_dir "/home/lbakken/development/rabbitmq/rabbitmq-server/plugins" -rabbit plugins_expand_dir "/tmp/rabbitmq-test-instances/rabbit@shostakovich/plugins" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/tmp/rabbitmq-test-instances/rabbit@shostakovich/mnesia/rabbit@shostakovich" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672

Testing Windows next.

michaelklishin · 2018-02-23T20:22:15Z

I can confirm that @lukebakken's changes make the issue go away. I was also able to config SCHEDULER_BIND_TYPE and a few other things via rabbitmq-env.conf.

Doing more testing next.

lukebakken · 2018-02-23T20:47:06Z

Tested on Windows 8.1 using this alpha build with all of the scripts/*.bat files copied from this branch into sbin. After doing the copy, you must run rabbitmq-service.bat remove then rabbitmq-service.bat install for erlsrv.exe to be installed with the correct arguments for erl.exe. Contents of %AppData%\RabbitMQ\rabbitmq-conf-env.bat:

set SCHEDULER_BIND_TYPE=u
set DISTRIBUTION_BUFFER_SIZE=65536
set MAX_NUMBER_OF_PROCESSES=131072
set MAX_NUMBER_OF_ATOMS=196608

gerhard · 2018-02-23T21:56:05Z

We used to have the RABBITMQ_ prefix since #873. The sudo logic used to be:

if RABBITMQ_SCHEDULER_BIND_TYPE env is set
then use this value in SERVER_ERL_ARGS
otherwise set RABBITMQ_SCHEDULER_BIND_TYPE to DEFAULT_SCHEDULER_BIND_TYPE

Was this incorrect?

michaelklishin · 2018-02-23T22:18:51Z

@gerhard the docs suggest rabbitmq-env.conf should contain var, which takes precedence over DEFAULT_var. Both are overriden by RABBITMQ_var from the environment should it be set. This logic has been around for years. #873 therefore is incomplete.

gerhard · 2018-02-26T09:13:14Z

@michaelklishin got it. So we basically need to support both RABBITMQ_var_name & var_name defined in rabbitmq-env.conf. I will update the tests to cover both cases.

Covers the following variables: * RABBITMQ_SCHEDULER_BIND_TYPE & SCHEDULER_BIND_TYPE * RABBITMQ_DISTRIBUTION_BUFFER_SIZE & DISTRIBUTION_BUFFER_SIZE * RABBITMQ_MAX_NUMBER_OF_PROCESSES & MAX_NUMBER_OF_PROCESSES * RABBITMQ_MAX_NUMBER_OF_ATOMS & MAX_NUMBER_OF_ATOMS

gerhard · 2018-02-26T12:08:42Z

@michaelklishin should be ready to merge.

gerhard added 2 commits February 23, 2018 18:05

Make maximum number of Erlang processes configurable via env

2739b40

This property is essential for nodes that need to handle many connections / channels / queues. Related to #1513

gerhard requested review from michaelklishin, dumbbell and lukebakken February 23, 2018 18:06

gerhard changed the title ~~Configurable max procs max atoms~~ Maximum number of Erlang processes and atoms can be configured via environment variables Feb 23, 2018

michaelklishin mentioned this pull request Feb 23, 2018

Configurable max procs max atoms master #1529

Merged

lukebakken previously approved these changes Feb 23, 2018

View reviewed changes

michaelklishin approved these changes Feb 23, 2018

View reviewed changes

michaelklishin requested changes Feb 23, 2018

View reviewed changes

michaelklishin reviewed Feb 23, 2018

View reviewed changes

Variables in rabbitmq-env.conf and rabbitmq-env-conf.bat do not begin…

c4b1f36

… with RABBITMQ_, adjust Unix and Windows files accordingly

lukebakken self-requested a review February 23, 2018 19:51

Ensure SERVER_ERL_ARGS are set on Windows

ff5d55a

lukebakken approved these changes Feb 23, 2018

View reviewed changes

michaelklishin approved these changes Feb 23, 2018

View reviewed changes

michaelklishin mentioned this pull request Feb 23, 2018

Tune scheduling flags for Erlang VM #612

Closed

Fix default maximum number of Erlang processes

512a57b

michaelklishin approved these changes Feb 26, 2018

View reviewed changes

michaelklishin merged commit 631f822 into v3.7.x Feb 26, 2018

lukebakken deleted the configurable-max-procs-max-atoms branch February 26, 2018 15:28

michaelklishin added this to the 3.7.4 milestone Feb 26, 2018

Maximum number of Erlang processes and atoms can be configured via environment variables #1528

Maximum number of Erlang processes and atoms can be configured via environment variables #1528

Uh oh!

Conversation

gerhard commented Feb 23, 2018

Proposed Changes

Types of Changes

Checklist

Uh oh!

gerhard commented Feb 23, 2018

Uh oh!

michaelklishin commented Feb 23, 2018

Uh oh!

lukebakken commented Feb 23, 2018

Uh oh!

michaelklishin commented Feb 23, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gerhard commented Feb 23, 2018

Uh oh!

gerhard commented Feb 23, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

michaelklishin left a comment

Choose a reason for hiding this comment

Uh oh!

michaelklishin Feb 23, 2018

Choose a reason for hiding this comment

Uh oh!

gerhard Feb 23, 2018

Choose a reason for hiding this comment

Uh oh!

lukebakken commented Feb 23, 2018

Uh oh!

michaelklishin commented Feb 23, 2018

Uh oh!

lukebakken commented Feb 23, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gerhard commented Feb 23, 2018

Uh oh!

michaelklishin commented Feb 23, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gerhard commented Feb 26, 2018

Uh oh!

gerhard commented Feb 26, 2018

Uh oh!

Uh oh!

michaelklishin commented Feb 23, 2018 •

edited

Loading

gerhard commented Feb 23, 2018 •

edited

Loading

lukebakken commented Feb 23, 2018 •

edited

Loading

michaelklishin commented Feb 23, 2018 •

edited

Loading