Replace opendistro

[sjpb]

Ticket: https://stackhpc.atlassian.net/browse/DEV-855

OpenDistro is EOL.

This PR:

• Replaces OpenDistro with OpenSearch.

• Updates filebeat to the newest-supported version.

• Adds the required version faking to enable filebeat.

• Configures important opensearch settings for production use.

• Updates Grafana version

• Updates the opensearch Grafana datasource plugin definition

• Removes the appliances grafana-datasources role as we can use grafana's provisioning mode with the cloudalchemy.grafana rather than requiring a customised API-based approach. <-- TODO CHECK: think this was merged already.

• Adds a test in CI that expected jobs from the hpctests runs are found via Grafana (NB: for slurm-stats this has to be 5 mins past job completion, so may add some delay)

• Changes the storage used for open{distro,search} from a podman volume to a host directory to enable easier future upgrades/migration/backups.

• Adds a playbook ansible/adhoc/migrate-opendistro.yml to migrate opendistro data to opensearch (checked by upgrading a running cluster from main 7bcacb0)

• Uses a new "prebuilt" image in arcus with the updated Grafana version (note actually CI was using this before this PR, so grafana's been getting downgraded during CI deployments)

• Merge workaround for ohpc-base-compute dependency on singularity.

• Update image build with correct grafana version, in a PR, and test that.

• TODO: fix/document migration - currently it will always run if opendistro service even exists.

• [ ]
  Once merged and passed:

• Move appropriate image to release bucket

Closes #70.

[sjpb]

It is leaving massive gaps between jobs:

ETA: fixed by 3a25308

[sjpb]

3a25308 fails on checking that the hpctests jobs exist in grafana/opensearch. Turns out neither dashboard nor datasources have been provisioned in grafana after rebuilding control node with packer-built image (although grafana is running). Possibly this has always been broken, just not checked for until this PR.

direct configuration:

2022-08-11T12:48:19.5059996Z TASK [cloudalchemy.grafana : Create/Update datasources file (provisioning)] ****
2022-08-11T12:48:19.5062754Z task path: /home/runner/work/ansible-slurm-appliance/ansible-slurm-appliance/ansible/roles/cloudalchemy.grafana/tasks/datasources.yml:26
2022-08-11T12:48:23.6044928Z NOTIFIED HANDLER cloudalchemy.grafana : restart grafana for ci2839370468-control
2022-08-11T12:48:23.6047794Z changed: [ci2839370468-control] => {
2022-08-11T12:48:23.6048876Z     "changed": true,
2022-08-11T12:48:23.6050009Z     "checksum": "c3246446b05c316d4a9dfde60badfa068fd3936c",
2022-08-11T12:48:23.6051407Z     "dest": "/etc/grafana/provisioning/datasources/ansible.yml",
2022-08-11T12:48:23.6052598Z     "gid": 979,
2022-08-11T12:48:23.6053560Z     "group": "grafana",
2022-08-11T12:48:23.6054671Z     "md5sum": "ed1ebc1e5e73ffca64f835dea1145202",
2022-08-11T12:48:23.6055784Z     "mode": "0640",
2022-08-11T12:48:23.6056768Z     "owner": "root",
2022-08-11T12:48:23.6057824Z     "secontext": "system_u:object_r:etc_t:s0",
2022-08-11T12:48:23.6058901Z     "size": 569,
2022-08-11T12:48:23.6060242Z     "src": "/var/lib/rocky/.ansible/tmp/ansible-tmp-1660222100.111777-5114-195439064785952/source",
2022-08-11T12:48:23.6061512Z     "state": "file",
2022-08-11T12:48:23.6062449Z     "uid": 0
2022-08-11T12:48:23.6063363Z }

control image build:

2022-08-11T12:57:56.6329342Z     openstack.control: TASK [cloudalchemy.grafana : Create/Update datasources file (provisioning)] ****
2022-08-11T12:57:56.6339799Z     openstack.control: task path: /home/runner/work/ansible-slurm-appliance/ansible-slurm-appliance/ansible/roles/cloudalchemy.grafana/tasks/datasources.yml:26
2022-08-11T12:57:56.6530631Z     openstack.control: skipping: [default] => {
2022-08-11T12:57:56.6532331Z     openstack.control:     "changed": false,
2022-08-11T12:57:56.6533879Z     openstack.control:     "skip_reason": "Conditional result was False"
2022-08-11T12:57:56.6535269Z     openstack.control: }
2022-08-11T12:57:56.7564322Z     openstack.control:```

[sjpb]

Note certs have a hardcoded 2yr life.

[sjpb]

@m-bull I tried using community.crypto.x509_certificate_info to extract validity and delete if necessary, but as podman chowns everything in certs/ the ansible loops/logic were just getting really messy. Put that on hold as not best use of effort at the moment.

[sjpb]

FIXED: that merge won't be right as we need an image using updated grafana etc.