Use bootstrap tokens provisioned by ansible for K3s instead of persistent tokens in cloud-init metadata

[wtripp180901]

Server now generates a short lived bootstrap token which nodes use for initial connection/auth. Performed in site.yml rather than ansible-init. Tokens no longer needs to be generated by ansible, but consistent node passwords are needed (previously we were using the ansible-generated token for this) so added to the secret store.

NB: When upgrading previous clusters, the groupsk3s_server and k3s_agent must now be defined - see environments/common/layouts/everything.

NB: Given k3s is not currently required, it could be disabled for production clusters (even when using a stackhpc image) by removing any hosts from the k3s_server and k3s_agent groups.

[wtripp180901]

https://github.com/stackhpc/ansible-slurm-appliance/actions/runs/13409545021

[sd109]

@wtripp180901 Do we also need to update this table in the compute-init README as a result of this change?

Do we have a plan yet for how to get nodes to rejoin the k3s cluster after a compute-init driven rebuild? Is it even possible with this bootstrap token approach?

[sjpb]

@sd109

Do we have a plan yet for how to get nodes to rejoin the k3s cluster after a compute-init driven rebuild? Is it even possible with this bootstrap token approach?

Yeah I think so, we discussed it the other day. The RebootProgram will have to generate a new token, that will be distributed by 🪄 (i.e. however we distribute other secrets), then the node picks it up on boot.

A failure to provision a new token which results in RebootProgram failing and node being marked down (unless we do something to handle that) so I think that will work.

[wtripp180901]

https://github.com/stackhpc/ansible-slurm-appliance/actions/runs/13453455728

[sjpb]

Still want to change token definition.

[sjpb]

Tiiny comments, one on security

[sjpb]

LGTM