Enable build of environment-specific control images

[sjpb]

Enables building environment-specific control images in the same way as for login/compute builds.

Ticket: https://stackhpc.atlassian.net/browse/DEV-695

Note this does NOT move state off-board from the controller, so reimaging the controller will loose all state. However it does at least provide an image which can be used to create a working controller for an existing cluster (regardless of e.g. upstream package changes).

Note that ansible/site.yml will need running after imaging a node with this image, to set:

• Partition information in slurm.conf - even if compute nodes are up during image build they are not in the packer play so this information cannot be templated
• Potentially Prometheus scrape config, depending on which compute nodes were up during image build

Fixes #133. Replaces #136.

Requires:

• openhpc role bump & dependency pinning #159
• Allow no partition definitions ansible-role-openhpc#132
• Update requirements.yml
• Packer build fixes #177

Note that with openondemand enabled, building a control image BEFORE deploying the login node fails in smslabs environment as grafana needs to know the openondemand_servername which requires the private IP (defined as .ansible_host) for the login node. This doesn't occur in CI as a direct deployment of control/login/2x compute is done first which generates the hosts inventory file with this information.

This limitation could be fixed in a later PR / for other environments by changing the TF to be two stages:

1. Deploy ports to get private IPs and template hosts file using this info (instead of from instances).
2. Deploy instances
   and only doing step 1 before running the image build. That should also allow login & compute image build without the control node actually existing.

CI does not currently try the built image.

NB: CI is failing until requirements.yml updated, but waiting for another PR to merge on openhpc role before bumping version

Dev deployment: vglabs-steveb-ansible-rocky85:/home/rocky/slurm-app-control-images

[sjpb]

Currently failing on:

    openstack.control: TASK [stackhpc.openhpc : Template basic slurm.conf] ****************************
    openstack.control: task path: /home/runner/work/ansible-slurm-appliance/ansible-slurm-appliance/ansible/roles/stackhpc.openhpc/tasks/runtime.yml:84
    openstack.control: An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ansible.errors.AnsibleFilterError: Group "ci2064964918_small" contains no hosts in this play - was --limit used?

Didn't see this in dev deployment as that had no group defined!

[sjpb]

Think CI ran out of instances, can retry later.

[sjpb]

@m-bull I've added 91e91e5 to this, to skip block_devices during build. The need for that commit only surfaced as part of #173. However we are using block devices to manage volumes for /home and stackhpc.openhpc:openhpc_state_save_location (which was already in before #173) in e.g. AlaSKA - both on the control node. So adding this commit to the PR will at least avoid control image build crashing on a deployed configuration.