Expose alarms as Prometheus metrics

[deadtrickster]

Proposed Changes

This exposes alarms as Prometheus metrics

Types of Changes

What types of changes does your code introduce to this project?
Put an x in the boxes that apply

• Bug fix (non-breaking change which fixes issue #NNNN)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause an observable behavior change in existing systems)
• Documentation improvements (corrections, new content, etc)
• Cosmetic change (whitespace, formatting, etc)

Checklist

Put an x in the boxes that apply. You can also fill these out after creating
the PR. If you're unsure about any of them, don't hesitate to ask on the
mailing list. We're here to help! This is simply a reminder of what we are
going to look for before merging your code.

• I have read the CONTRIBUTING.md document
• I have signed the CA (see https://cla.pivotal.io/sign/rabbitmq)
• All tests pass locally with my changes
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)
• Any dependent changes have been merged and published in related repositories

Further Comments

Closes #2653

[michaelklishin]

This works as expected but I think we can do better than "X_limit" for alarm names. Except for, perhaps, the file descriptor alarm, they do not represent limits but just watermarks.

Disk space can continue to decreased after the configured (low) watermark is crossed.

[deadtrickster]

With names I was following the alarms names. Really straightforward mapping: {resource_limit, disk, _} -> rabbitmq_alarms_disk_limit.
That being said, these pages https://www.rabbitmq.com/memory.html and https://www.rabbitmq.com/disk-alarms.html point towards some inconsistency in naming - namely vm_memory_high_watermark.relative = 0.4 vs disk_free_limit.absolute = 1000000000.
Now I wonder if we should have these 3:

• rabbitmq_alarms_file_descriptor_limit
• rabbitmq_alarms_vm_memory_high_watermark
• rabbitmq_alarms_disk_free_limit.

I.e. follow rabbitmqctl/config conventions?

[michaelklishin]

There is no convention around configuration setting names: they were introduced at different points without much focus on making them consistent.

I kept file_descriptor_limit as is because it's the most precise name I could think of and changed the other two to say "watermark" instead of "limit" because they both are watermarks/thresholds.

[michaelklishin]

@Mergifyio backport v3.9.x v3.8.x

[mergify]

Command backport v3.9.x v3.8.x: success

Backports have been created

• #3315 Expose alarms as Prometheus metrics (backport #3312) has been created for branch v3.9.x
• #3316 Expose alarms as Prometheus metrics (backport #3312) has been created for branch v3.8.x

[michaelklishin]

Backported manually to v3.8.x for 3.8.25.