Add "vm_memory_high_watermark" to the generated config, sourced from the value of the "memory.limit_in_bytes" cgroup restriction

[tianon]

Closes #48

Some example test cases:

$ docker run -it --rm 842aed1a0ec9
...
Memory limit set to 31616MB of 31616MB total.
...
$ docker run -it --rm -m 100m 842aed1a0ec9
...
Memory limit set to 100MB of 31616MB total.
...

I'm not sure if we should be going slightly lower than the available value.

I checked the standard default for this output when vm_memory_high_watermark isn't specified, and got the following:

$ docker run -it --rm rabbitmq
...
Memory limit set to 12646MB of 31616MB total.
...

So it looks like the default behavior is actually a fraction of the total available.

[tianon]

(the other minor changes in this PR just make sure that this value is actually set appropriately even if no environment variable configuration is supplied)

[tianon]

root@08883e8f3ead:/# cat /etc/rabbitmq/rabbitmq.config 
[ { rabbit, [
    { loopback_users, [ ] },
    { vm_memory_high_watermark, { absolute, 104857600 } },
    { tcp_listeners, [ 5672 ] },
    { ssl_listeners, [ ] },
    { hipe_compile, false }
] } ].

[yosifkit]

Maybe we shouldn't set it at all if the memory.limit_in_bytes is higher than total memory, so that we get the default of about 40% of total memory. That should prevent users from accidentally handing all their host memory to rabbitmq. 😮

[tianon]

https://www.rabbitmq.com/production-checklist.html#resource-limits-ram has some good info about recommendations for setting this value

I'm wondering if we should only do something like 90% of the available cgroup limit, or just subtract some static overhead (perhaps based on the current usage at the time we calculate the value?)

[yosifkit]

Since rabbitmq (and most programs with memory limits like java) don't take into account the cgroup limit, we could allow passing in a value for the vm_memory_high_watermark.

• [0.0-1.0] percent of RAM to use
  • if there is a cgroup limit, calculate the number of byes and give bytes to rabbitmq
  • no cgroup limit, just pass the percent to rabbitmq
• [int] number of byes to use
  • just pass it to rabbit (maybe sanity check for >= 128MB and less than cgroup limit)
• [string] ex 200MiB
  • add some quote to the generated output
• no value supplied
  • if there is a cgroup limit, default to 90% of it? (maybe 85 or 80)
  • just let rabbit use its 40% default

[jperville]

Hello, any chance to have this PR merged (after rebasing)? Having properly configured high memory watermark when using memory limit is important feature to us.

After backporting the feature to my own RabbitMQ docker image, I found that setting the value of vm_memory_high_watermark to 100% of the value of memory.limit_in_bytes is too high because it does not leave any room for Erlang GC to do its thing. Rather I set vm_memory_high_watermark to 40% of the value of memory.limit_in_bytes as recommended in https://www.rabbitmq.com/memory.html

[tianon]

Thanks for the prod, @jperville ❤️

I've updated this to implement something similar to what @yosifkit laid out above:

• if we have no limit and no RABBITMQ_VM_MEMORY_HIGH_WATERMARK, don't write anything in config (let default 40% ride)
• if we have a cgroup limit smaller than MemTotal and no RABBITMQ_VM_MEMORY_HIGH_WATERMARK, use 40% of the cgroup limit
• otherwise, interpret the value of RABBITMQ_VM_MEMORY_HIGH_WATERMARK in the following ways:
  • NN% converted to 0.NN (ie, 50% becomes 0.5)
  • 0.NN (ie, 0.5) is a percentage; if we have a cgroup limit, perform the math to use { absolute, xxxx } appropriately, otherwise just embed the percentage value directly and let RabbitMQ perform the math
  • NNNNN (ie, 10240000) is raw bytes, { absolute, NNNNN }
  • NNNNNsuffix (ie, 1024MiB) is stringified bytes, { absolute, "NNNNNsuffix" }

[jperville]

How long does it take for the official image on docker hub to be updated with the new entrypoint? Still not updated 15 hours after.

[tianon]

We'll make a PR to update the library later today. 👍

From https://github.com/docker-library/rabbitmq/blob/master/README.md:

See a change merged here that doesn't show up on the Docker Hub yet? Check the "library/rabbitmq" manifest file in the docker-library/official-images repo, especially PRs with the "library/rabbitmq" label on that repo. For more information about the official images process, see the docker-library/official-images readme.

[qiuyuzhou]

The official image is failing for me on memory allocation "Memory limit set to 0MB of 16049MB total."
After specify the env variable RABBITMQ_VM_MEMORY_HIGH_WATERMARK=300MiB then it work.

[AnatolyRugalev]

I have the same issue as @qiuyuzhou

Setting RABBITMQ_VM_MEMORY_HIGH_WATERMARK to 200MiB helped

[qiuyuzhou]

  rabbitmq:
    hostname: rabbitmq
    networks:
      - sd-backend
    image: rabbitmq:3-alpine
    environment:
      - RABBITMQ_DEFAULT_USER=guest
      - RABBITMQ_DEFAULT_PASS=guest
      #- RABBITMQ_VM_MEMORY_HIGH_WATERMARK=300MiB
    volumes:
      - /var/lib/rabbitmq

Without specify RABBITMQ_VM_MEMORY_HIGH_WATERMARK, then it failed.

[AnatolyRugalev]

I think we could add information about this var to README at least

[tianon]

Docs PR is up at docker-library/docs#879. 👍

[tianon]

I do want to repeat my request from #147 (comment) though:

Can one of those who is getting Memory limit set to 0MB ... please give the full output of docker run --rm alpine cat /sys/fs/cgroup/memory/memory.limit_in_bytes /proc/meminfo ?

I've got #149, but I can't reproduce whatever is causing 0MB to be the detected value in the first place, so I want to check my assumptions. 😰

[jmaroeder]

@tianon Just ran into this error. Here's my output:

18446744073709551615
MemTotal:       15400952 kB
MemFree:         3555028 kB
Buffers:          336116 kB
Cached:          3525148 kB
SwapCached:            0 kB
Active:          8663360 kB
Inactive:        1906480 kB
Active(anon):    6713496 kB
Inactive(anon):    18964 kB
Active(file):    1949864 kB
Inactive(file):  1887516 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:             39296 kB
Writeback:             0 kB
AnonPages:       6708388 kB
Mapped:           102876 kB
Shmem:             23880 kB
Slab:            1037464 kB
SReclaimable:     846732 kB
SUnreclaim:       190732 kB
KernelStack:       17448 kB
PageTables:        65244 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     7700476 kB
Committed_AS:   18249600 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       38952 kB
VmallocChunk:   34359569076 kB
HardwareCorrupted:     0 kB
AnonHugePages:   4323328 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       79872 kB
DirectMap2M:     5294080 kB
DirectMap1G:    10485760 kB

[tianon]

Ahhh, thank you @jamesmallen -- that explains it. That 18446744073709551615 is too large for Bash to handle!

$ echo $(( 18446744073709551615 / 1024 ))
0
$ echo $(( 18446744073709551615 * 40 / 100 ))
0

Going to update #149 now! ❤️ ❤️

[tianon]

Compared to awk:

$ awk 'BEGIN { v = 18446744073709551615; print v / 1024 }'
18014398509481984
$ awk 'BEGIN { v = 18446744073709551615; print v * 0.4 }'
7378697629483821056