Add support for alertmanager

[sjpb]

• Adds alertmanager:
  • On by default, requires enabling slack integration to get any alerts
  • Not on for caas
  • No HA
  • No persistent state
  • basic_auth on web interface
    (see ansible/roles/alertmanager/README.md for more)
• Adds default alert rules for node exporter and slurm nodes down or failed
• Fixes a bug where prometheus env label was not set from hostvars. Now can be set using host/group var prometheus_env to group nodes by e.g. rack
• Adds prometheus label group to node_exporter targets, set to login, control, compute, or other to enable targeting alerting rules.
• Fixes a bug where compute-init could fail on boot due to json-encoding of binary munge key

[sjpb]

Note tvncserver is falling over on all nodes. TBH I wasn't expecting it to be enabled/started unless a job requested it. Should fix.

[sjpb]

trying running ondemand remote desktop with service stopped:

[root@RL9-compute-0 rocky]# systemctl stop tvncserver
[root@RL9-compute-0 rocky]# systemctl disable tvncserver
tvncserver.service is not a native service, redirecting to systemd-sysv-install.
Executing: /usr/lib/systemd/systemd-sysv-install disable tvncserver

worked ok

[sjpb]

Image build: https://github.com/stackhpc/ansible-slurm-appliance/actions/runs/14465582790

[sjpb]

Image build: https://github.com/stackhpc/ansible-slurm-appliance/actions/runs/14491874938

[wtripp180901]

Looks good, mostly just configurability/consistency changes.

Also is there a reason there's an empty task file in the alertmanager role?

[wtripp180901]

LGTM

[sjpb]

Image build: https://github.com/stackhpc/ansible-slurm-appliance/actions/runs/14595938710

[sjpb]

Image build: https://github.com/stackhpc/ansible-slurm-appliance/actions/runs/14595969709

[wtripp180901]

LGTM