RFC rabbit_amqqueue_process: improve message duplicates handling

[noxdafox]

As discussed in #1741, this patch series take into use the newly defined is_duplicate behaviour in rabbitmq/rabbitmq-common#283.

Conversely to #1741, this series should not be changing the original behaviour but rather extending it.

If the message is duplicated due to an internal error, the backing queue detecting it can instruct the amqqueue process to simply ignore it.
If the message is instead deemed a duplicate, the publisher will be notified with a rejection.

Types of Changes

• Bugfix (non-breaking change which fixes issue #NNNN)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation (correction or otherwise)
• Cosmetics (whitespace, appearance)

Checklist

• I have read the CONTRIBUTING.md document
• I have signed the CA (see https://cla.pivotal.io/sign/rabbitmq)
• All tests pass locally with my changes
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)
• Any dependent changes have been merged and published in related repositories

Note on tests:

There were few tests failing with the latest changes in master in the cluster_rename_SUITE test suite. This patch series did not cause any new test to fail.

[michaelklishin]

This looks good after a quick review. Have you done any long-running benchmark to assess the impact of this, in particular on priority queues? Your changes seem roughly equivalent but this is a pretty sensitive code path.

Thank you for your ongoing contributions!

[noxdafox]

Valid point. I will try some long running tests with HA and priority queues enabled.

We'll also need to change the management console message as now states: "rejected Unable to publish message. Check queue limits."

Now there are more cases where a message might not reach the queue than a mere queue overflow.

[michaelklishin]

I'm not sure what message that is but it can be discussed in a separate management plugin PR.

[noxdafox]

Got time to run some tests.

Successfully ran several rounds with the following set up.

• 10 queues
• 10 priority levels per queue
• 1 producer and 1 consumer per queue
• Each producer publishing 10K messages per priority for a total of 100K messages per queue
• Each consumer ensuring 100K messages are received correctly

I did not encounter any issue. All queues were constantly empty and broker messages per second kept between 2.5K to 3K.

I also run several tests with the above set up and the message deduplication plugin enabled to test the rejection flow. Did not observe any issue, performance close to the above mentioned. Messages were de-duplicated correctly.

Any other scenario you think I should test?

[michaelklishin]

@noxdafox thank you, that sounds like a good start. I will QA this and then test a few workflows we use on our team.