Support configure max sync throughput in CMQs

[thuandb]

Proposed Changes

We would like to propose a backward compatible change in order to reduce memory utilization during classic mirrored queue sync.

Specifically, in addition to the existing mirroring_sync_batch_size configuration key, which controls maximum number of messages per sync batch, we propose to add a new configuration key mirroring_sync_max_throughput in bytes per second to control maximum synchronization throughput. The syncer at the primary node will make sure the throughput of messages being synchronized does not exceed the value configured. This will give mirroring nodes sufficient time to page messages being synchronized to disk and quickly free up memory.

The motivation for this is that, in our test with a 3-node cluster on AWS with data volumes using EBS gp2 (maximum disk write throughput is 250MiB/s), we observe that the memory utilization during queue sync is high leading to a memory alarm. We suspect that when disk write throughput is low, mirroring nodes take time to page messages to disk while the syncer keeps broadcasting messages at a higher rate. As a result, most messages are kept in memory on the mirror(s) and are waiting to be paged to disk leading to high memory usage.

With this change, if the max sync throughput value is set to less than the write throughput of data volumes, (i.e., the primary node never broadcast data at a speed higher than disk write speed), it will help reduce memory utilization at mirroring node significantly. The new configuration will be disabled by default (mirroring_sync_max_throughput = 0) to ensure backward compatibility. I.e., nothing changes to existing/running RabbitMQ clusters. Users will have to explicitly set sync throughput to enable the feature.

Here are samples of memory utilization at the mirroring node and the primary node with different message sizes in tests using RabbitMQ 3.8.22 in a cluster with EBS gp2 volume compared with the limit sync throughput feature:

Message size of 640KiB

RabbitMQ 3.8.22

A test with RabbitMQ 3.8.22 using batch size of 128 shows memory consumption at mirroring node is very high (around memory alarm level); (memory utilization at the primary node is good however).

• Memory consumption during queue sync at the mirroring node is high

• Memory consumption during queue sync at the primary node looks good.

Limit sync throughput

We tested this change with mirroring_sync_max_throughput = 150MiB with similar setting, memory consumption is low at both mirroring node and primary node.

• Memory consumption during queue sync at the mirroring node:

• Memory consumption during queue sync at the primary node is low too

Message size of 10MiB

A test with RabbitMQ 3.8.22 using batch size of 128 shows memory utilization is very high at both mirroring node and primary node.

RabbitMQ 3.8.22

• Memory consumption during queue sync at the mirroring node is high

• Memory consumption during queue sync at the primary node is high too.

Limit sync throughput

We tested with mirroring_sync_max_throughput = 150MiB, memory consumption drops significantly at both mirroring node and primary node.

• Memory consumption during queue sync at the mirroring node dropped significantly.

• Memory consumption during queue sync at the primary node is low as well

We also conducted tests with various settings such as using default batch size of 4096 as well as with smaller message sizes; they all show that when sync throughput is limited to a lower level of disk write throughput at mirroring nodes, memory consumed by queue sync processes significantly improves.

Types of Changes

What types of changes does your code introduce to this project?
Put an x in the boxes that apply

• Bug fix (non-breaking change which fixes issue #NNNN)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause an observable behavior change in existing systems)
• Documentation improvements (corrections, new content, etc)
• Cosmetic change (whitespace, formatting, etc)
• Build system and/or CI

Checklist

Put an x in the boxes that apply.
You can also fill these out after creating the PR.
If you're unsure about any of them, don't hesitate to ask on the mailing list.
We're here to help!
This is simply a reminder of what we are going to look for before merging your code.

• I have read the CONTRIBUTING.md document
• I have signed the CA (see https://cla.pivotal.io/sign/rabbitmq)
• I have added tests that prove my fix is effective or that my feature works
• All tests pass locally with my changes
• If relevant, I have added necessary documentation to https://github.com/rabbitmq/rabbitmq-website

• We will submit a separate PR for updating doc once this change is approved/merged.

• If relevant, I have added this change to the first version(s) in release-notes that I expect to introduce it

[michaelklishin]

I like how small the solution ended up being.