Merge caas slurm appliance into slurm appliance

[sjpb]

Replaces https://github.com/stackhpc/caas-slurm-appliance with a .caas environment.

Requires https://github.com/stackhpc/azimuth-caas-operator/pull/75 and azimuth-cloud/ansible-collection-azimuth-ops#202. Note that:

• ui-meta files are provided in environments/.caas/
• the cluster playbook is now ansible/site.yml
  Both normal and "fast-volume-type" clusters are supported, by using the appropriate ui-meta file.

Currently the ansible-driven image build functionality from CaaS has not been ported to this repo - use the github workflow or or packer configurations directly.

Implementation Notes

• There can't be an activate script, so:
  • ansible.cfg is provided in the repo root, as expected by caas operator.
  • ANSIBLE_INVENTORY is set in the cluster type template, using a relative path:

    azimuth_caas_stackhpc_slurm_appliance_template:
    ...
      envVars:
        # Normally set through environment's activate script:
        ANSIBLE_INVENTORY: environments/common/inventory,environments/.caas/inventory # NB: Relative to runner project dir
    

    Ansible then defines ansible_inventory_sources which contains absolute paths, and that is used to derive the appliances_environment_root. This needed some changes in site.yml where the environment var APPLIANCES_ENVIRONMENT_ROOT is not available in caas. Note this environment var can't be got rid of entirely yet because {TF,PKR}_VAR_environment_root are derived from it in non-caas environments).
• Ansible-driven TF imported only used for caas at present.
• Inventory groups are now defined by symlink to everything template, and don't need to be created in-memory. This means groups always exist, so logic changes from if 'grafana' in groups --> if groups[‘grafana’] | length > 0 which means lots of caas-specific overrides could be removed.
• common env ondemand config didn’t always take into account whether grafana was deployed, which caas did (see 2b264c7).
• Conversely I think the caas ondemand config was broken for jupyter - wrong binary path.
• Contents of caas repo are now here as follows
  • caas slurm-infra.yml → pre.yml, site.yml, post.yml
  • caas roles/* → ansible/roles/
  • image_build{,_infra} - not currently ported
  • caas ui_meta → environments/.caas/ui_meta
  • caas ansible.cfg → ansible.cfg
  • caas group_vars → environments/.caas/inventory/group_vars
  • caas assets/* → environments/.caas/assets
  • requirements.yml → merged with requirements.yml
• basic_users and etc_hosts roles are used instead of custom code in the original caas repo.

TODO

Merge following from CaaS:

• Add restart delay for Zenith client units caas-slurm-appliance#44
• Note rocky user has passwordless sudo in usage message caas-slurm-appliance#43
• Bump zenith proxy client and mitm image to v0.1.0 caas-slurm-appliance#45

[sjpb]

Tests @ b78f1e4

• Checked azimuth can still deploy workstation: OK

On azimuth cluster - sb-tf1:

• Create with hpctests: OK
• direct ssh: OK
• OOD portal OK
• OOD shell: OK
• OOD jupyter: OK
• OOD desktop
• monitoring: OK
  • job list: OK
  • job details OK
• Delete: OK

Tests on stackhpc environment:

• provision: OK
• configure: OK
• hpctests: OK
• direct ssh: not tested
• OOD portal: OK
• OOD shell: OK
• OOD jupyter: OK
• OOD desktop: OK
• monitoring: OK
  • can see jobs
  • can see job details