Compute-Init: wait for cloud-init before NFS mount

We are seeing issues where compute-init hits:

TASK [Check if hostvars exist]
FAILED! => {"changed": false, "msg": "Permission denied"}

We have found we are ignoring errors on the mount.
Its possible the mount will fail if the host networking
has not be setup.

Lets wait to make sure we can talk to NFS before attempting
the NFS mount, mostly checking because the host networking
stack might not yet be setup correctly.

We could do "cloud-init status --wait" and block on cloud-init
having finished, however we don't really depend on all parts
of cloud-init being complete.
Equally, we could think about ansible-init systemd unit file
depending on cloud-init or the network being available, but
there are cases where we do not want that.

Author: JohnGarbutt

ansible/roles/compute_init/files/compute-init.yml

@@ -61,7 +61,13 @@
         owner: slurm
         group: root
         mode: u=rX,g=rwX,o=
-    
+
+    - name: Wait for NFS to reachable (checks host network up)
+      ansible.builtin.wait_for:
+        port: 2049
+        host: '{{ server_node_ip }}'
+        timeout: 120
+
     - name: Mount /mnt/cluster
       mount:
         path: /mnt/cluster
@@ -70,8 +76,6 @@
         opts: ro,sync
         state: mounted
       register: _mount_mnt_cluster
-      ignore_errors: true
-      # TODO: add some retries here?
 
     - block:
         - name: Report skipping initialization if cannot mount nfs