fix: use rabbitmq length for RabbitMQNodeDown (#1579)

The `RabbitMQNodeDown` made the assumption that all deployments involve
only three RabbitMQ nodes. However, this is not always the case as we
do support deployments with a single node or more than three.

Before this would have caused false alerts in deployments with a single
RabbitMQ node. Whilst also concealing alerts in deployments with more
than three nodes.

Co-authored-by: Matt Crees <[email protected]>

Author: jackhodgkiss

etc/kayobe/kolla/config/prometheus/rabbitmq.rules

@@ -6,7 +6,7 @@ groups:
 - name: rabbitmq.rules
   rules:
   - alert: RabbitMQNodeDown
-    expr: sum(rabbitmq_build_info{instance!=""}) < 3
+    expr: sum(rabbitmq_build_info{instance!=""}) < {% endraw %}{{ alertmanager_number_of_rabbitmq_nodes }}{% raw %}
     for: 30m
     labels:
       severity: critical

etc/kayobe/stackhpc-monitoring.yml

@@ -26,6 +26,9 @@ alertmanager_packet_drop_threshold: 1
 # packets/s averaged over 5 minutes.
 alertmanager_packet_errors_threshold: 1
 
+# Number of RabbitMQ nodes in the cluster.
+alertmanager_number_of_rabbitmq_nodes: "{{ groups['controllers'] | length }}"
+
 ###############################################################################
 # Exporter configuration
 

releasenotes/notes/use-length-for-rabbitmq-node-down-rule-c9e9c6b09f57954d.yaml

@@ -0,0 +1,8 @@
+---
+features:
+  - |
+    Allow for easy customisation of the number of expected `RabbitMQ`
+    nodes when evaluating the alert `RabbitMQNodeDown`. It is set by
+    the `alertmanager_number_of_rabbitmq_nodes` which defaults to the
+    number of `controllers`. This is benefical for deployments that
+    do not use a standard three node setup.