Support ansible-init for remote collections #411

sjpb · 2024-07-16T15:21:38Z

Enables remote ansible collections to be configured via cloud-init metadata and run via the ansible-init role.

Note that:

Any pre hook can run immediately but bootstrap.yml is blocked until ansible-init has run on each node where it is installed.
As ansible-init requires network access to download the specified remote collections, clusters using this functionality must have usable networking on instance boot and e.g. must not require the resolv_conf and/or proxy roles from bootstrap.yml.

Add linux-ansible-init role to requirements.yml
Run linux-ansible-init role during build so it is installed in the image - requires changes to groups and plays
Update the ansible/roles/cluster_infra TF templates to take variables ansible_init_collections and azimuth_init_playbooks in the caas environment and define metadata: e.g. similar to this PR for the workstation: https://github.com/stackhpc/caas-workstation/pull/22/files
Use defaults in the cluster_infra role for the ansible_init_* template vars instead of "if defined" guards everwhere.
Modify site.yml to only run once ansible-init has completed, as detected via sentinel file, when relevant.
Resolve name and role/collection status for ansible-init
Build CI image
Bump CI image
Aside: look into this:

[WARNING]: Invalid characters were found in group names but not replaced, use
-vvvv to see details

bertiethorpe · 2024-07-18T14:45:19Z

https://github.com/stackhpc/ansible-slurm-appliance/actions/runs/9993667651

bertiethorpe · 2024-07-19T09:43:19Z

https://github.com/stackhpc/ansible-slurm-appliance/actions/runs/10006009159

bertiethorpe · 2024-07-22T09:29:52Z

https://github.com/stackhpc/ansible-slurm-appliance/actions/runs/10038259001

bertiethorpe · 2024-07-22T13:39:59Z

https://github.com/stackhpc/ansible-slurm-appliance/actions/runs/10041020461

undo groupname changes

bertiethorpe · 2024-07-23T09:20:36Z

https://github.com/stackhpc/ansible-slurm-appliance/actions/runs/10056246261

environments/.stackhpc/terraform/main.tf

ansible/bootstrap.yml

environments/.stackhpc/inventory/group_vars/all/bastion.yml

…-slurm-appliance into feat/linux-ansible-init

bertiethorpe · 2024-07-25T12:43:25Z

https://github.com/stackhpc/ansible-slurm-appliance/actions/runs/10094314001

…-slurm-appliance into feat/linux-ansible-init

bertiethorpe · 2024-07-25T16:03:59Z

https://github.com/stackhpc/ansible-slurm-appliance/actions/runs/10097513582

sjpb · 2024-08-06T15:14:36Z

Tested on a CaaS deployment using

# requirements.yml:
collections:
  - name: https://github.com/stackhpc/ansible-collection-azimuth-ops.git
    type: git
    version: feat/ansible-init-slurm

# .github/environments/arcus/inventory/group_vars/all/z_devel.yml:

infra_flavor_id: 79defe96-d6ea-4130-855f-c247bb08b6b9 #vm.ska.cpu.general.quarter

#azimuth_caas_stackhpc_slurm_appliance_git_version
azimuth_caas_stackhpc_slurm_appliance_image: db91ac07-5a40-4e78-ab69-3290b56d3322 # openhpc-ofed-RL9-240725-1710-325c7b47
admin_dashboard_ingress_basic_auth_password: <redacted>

# enable existing project manila:
azimuth_openstack_manila_project_share_gb: 40
azimuth_caas_stackhpc_slurm_appliance_home_manila_share: true
azimuth_caas_stackhpc_slurm_appliance_extra_vars_overrides:
  cluster_home_manila_share_type: ceph01_cephfs

# test ansible-init remote collections:
azimuth_caas_stackhpc_slurm_appliance_init_collections_extra:
  - type: git
    name: https://github.com/stackhpc/ansible-collection-test.git
    version: main
azimuth_caas_stackhpc_slurm_appliance_init_playbooks_extra: 
  - name: stackhpc.test_collection.test
    stage: pre

mkjpryor

LGTM

* Add ansible-init role to requirements.yml * Add ansible-init to groups and plays * Configure cluster_infra ansible-init metadata * Only run site.yml once ansible-init has completed * Wait for ansible init to finish before running bootstrap * revert to using cluster_infra metadata defaults * update image * revert sausage bastion changes * set ansible_init_wait as common var * use run_number as a shorter ID for CI * install ood apps in fatimage * add ood jupyter install to fatimage * bump image * jupyter_compute ood into fatimage * bump fatimage for jupyter_compute ood * Update stackhpc.yml * duplicate tuned inventory group name * Fix invalid group name for slurmci * Update stackhpc.yml undo groupname changes * slurmci group name warning * rm ood changes * bump fatimage * change azimuth collection in bootstrap * update azimuth image utils version * update requirements * Update bastion.yml * Use azimuth image utils collection for ansible-init * bump fatimage --------- Co-authored-by: bertie <[email protected]> Co-authored-by: bertiethorpe <[email protected]>

* add trivy image scanning * bump fatimage workflow to ubuntu 22.04 * make setup script work in CI TODO: FIXME * fix libguestfs install * run only 1x build per matrix entry, & only builds required * fix packer README * fix image download * Use shorter names for CI clusters (#415) * use run_number as a shorter ID for CI * slurmci group name warning * Revert "slurmci group name warning" - underscores not valid linux hostname and stripped in host, leading to slurmdbd config failure This reverts commit 61dfad6. --------- Co-authored-by: Bertie <[email protected]> * install ood apps in fatimage * add ood jupyter install to fatimage * jupyter_compute ood into fatimage * bump fatimage * allow items in compute mapping to have different keys e.g. only specify image_id for some compute groups (#412) * Support ansible-init for remote collections (#411) * Add ansible-init role to requirements.yml * Add ansible-init to groups and plays * Configure cluster_infra ansible-init metadata * Only run site.yml once ansible-init has completed * Wait for ansible init to finish before running bootstrap * revert to using cluster_infra metadata defaults * update image * revert sausage bastion changes * set ansible_init_wait as common var * use run_number as a shorter ID for CI * install ood apps in fatimage * add ood jupyter install to fatimage * bump image * jupyter_compute ood into fatimage * bump fatimage for jupyter_compute ood * Update stackhpc.yml * duplicate tuned inventory group name * Fix invalid group name for slurmci * Update stackhpc.yml undo groupname changes * slurmci group name warning * rm ood changes * bump fatimage * change azimuth collection in bootstrap * update azimuth image utils version * update requirements * Update bastion.yml * Use azimuth image utils collection for ansible-init * bump fatimage --------- Co-authored-by: bertie <[email protected]> Co-authored-by: bertiethorpe <[email protected]> * avoid python-openstackclient v7 due to rebuild bug (#420) * Update hpctests to obey UCX_NET_DEVICES when RoCE devices present (#421) * Turn off higher priority MPI net devices * Update pingmatrix.sh.j2 * Update pingmatrix.sh.j2 * Update pingpong.sh.j2 * Replace j2 comments with bash * Update pingpong.sh.j2 --------- Co-authored-by: Steve Brasier <[email protected]> * delete trivy scanned vulnerabilities * update grafana * bump image * Update environments/.stackhpc/hooks/post.yml Co-authored-by: Steve Brasier <[email protected]> * Update setup-env.sh --------- Co-authored-by: Bertie <[email protected]> Co-authored-by: bertiethorpe <[email protected]> Co-authored-by: bertiethorpe <[email protected]>

* Add ansible-init role to requirements.yml * Add ansible-init to groups and plays * Configure cluster_infra ansible-init metadata * Only run site.yml once ansible-init has completed * Wait for ansible init to finish before running bootstrap * revert to using cluster_infra metadata defaults * update image * revert sausage bastion changes * set ansible_init_wait as common var * use run_number as a shorter ID for CI * install ood apps in fatimage * add ood jupyter install to fatimage * bump image * jupyter_compute ood into fatimage * bump fatimage for jupyter_compute ood * Update stackhpc.yml * duplicate tuned inventory group name * Fix invalid group name for slurmci * Update stackhpc.yml undo groupname changes * slurmci group name warning * rm ood changes * bump fatimage * change azimuth collection in bootstrap * update azimuth image utils version * update requirements * Update bastion.yml * Use azimuth image utils collection for ansible-init * bump fatimage --------- Co-authored-by: bertie <[email protected]> Co-authored-by: bertiethorpe <[email protected]>

* add trivy image scanning * bump fatimage workflow to ubuntu 22.04 * make setup script work in CI TODO: FIXME * fix libguestfs install * run only 1x build per matrix entry, & only builds required * fix packer README * fix image download * Use shorter names for CI clusters (#415) * use run_number as a shorter ID for CI * slurmci group name warning * Revert "slurmci group name warning" - underscores not valid linux hostname and stripped in host, leading to slurmdbd config failure This reverts commit 61dfad6. --------- Co-authored-by: Bertie <[email protected]> * install ood apps in fatimage * add ood jupyter install to fatimage * jupyter_compute ood into fatimage * bump fatimage * allow items in compute mapping to have different keys e.g. only specify image_id for some compute groups (#412) * Support ansible-init for remote collections (#411) * Add ansible-init role to requirements.yml * Add ansible-init to groups and plays * Configure cluster_infra ansible-init metadata * Only run site.yml once ansible-init has completed * Wait for ansible init to finish before running bootstrap * revert to using cluster_infra metadata defaults * update image * revert sausage bastion changes * set ansible_init_wait as common var * use run_number as a shorter ID for CI * install ood apps in fatimage * add ood jupyter install to fatimage * bump image * jupyter_compute ood into fatimage * bump fatimage for jupyter_compute ood * Update stackhpc.yml * duplicate tuned inventory group name * Fix invalid group name for slurmci * Update stackhpc.yml undo groupname changes * slurmci group name warning * rm ood changes * bump fatimage * change azimuth collection in bootstrap * update azimuth image utils version * update requirements * Update bastion.yml * Use azimuth image utils collection for ansible-init * bump fatimage --------- Co-authored-by: bertie <[email protected]> Co-authored-by: bertiethorpe <[email protected]> * avoid python-openstackclient v7 due to rebuild bug (#420) * Update hpctests to obey UCX_NET_DEVICES when RoCE devices present (#421) * Turn off higher priority MPI net devices * Update pingmatrix.sh.j2 * Update pingmatrix.sh.j2 * Update pingpong.sh.j2 * Replace j2 comments with bash * Update pingpong.sh.j2 --------- Co-authored-by: Steve Brasier <[email protected]> * delete trivy scanned vulnerabilities * update grafana * bump image * Update environments/.stackhpc/hooks/post.yml Co-authored-by: Steve Brasier <[email protected]> * Update setup-env.sh --------- Co-authored-by: Bertie <[email protected]> Co-authored-by: bertiethorpe <[email protected]> Co-authored-by: bertiethorpe <[email protected]>

bertiethorpe added 6 commits July 16, 2024 14:01

Add ansible-init role to requirements.yml

dc6e27c

Add ansible-init to groups and plays

4e09a6d

Configure cluster_infra ansible-init metadata

fd023ec

Only run site.yml once ansible-init has completed

b6246c9

Wait for ansible init to finish before running bootstrap

ec432d3

Merge branch 'main' into feat/linux-ansible-init

7ca2431

revert to using cluster_infra metadata defaults

1c8626e

bertiethorpe added 6 commits July 19, 2024 11:32

update image

21c6aac

revert sausage bastion changes

cdb1593

set ansible_init_wait as common var

78d8872

use run_number as a shorter ID for CI

6ea5969

install ood apps in fatimage

057c453

add ood jupyter install to fatimage

bb9609a

bertiethorpe added 3 commits July 22, 2024 10:19

bump image

fb2482a

jupyter_compute ood into fatimage

3547927

bump fatimage for jupyter_compute ood

ec9c6a7

bertiethorpe and others added 6 commits July 22, 2024 15:41

Update stackhpc.yml

9ac18ef

duplicate tuned inventory group name

bbf96ef

Fix invalid group name for slurmci

7ac8049

Update stackhpc.yml

e7bdbcc

undo groupname changes

slurmci group name warning

aab9eab

rm ood changes

f183a6a

bertiethorpe and others added 3 commits July 23, 2024 10:27

bump fatimage

8037c54

put dashes in slurmci back

d021e56

Merge branch 'main' into feat/linux-ansible-init

8ff6d9e

sjpb commented Jul 23, 2024

View reviewed changes

environments/.stackhpc/terraform/main.tf Outdated Show resolved Hide resolved

sjpb commented Jul 25, 2024

View reviewed changes

ansible/bootstrap.yml Outdated Show resolved Hide resolved

sjpb commented Jul 25, 2024

View reviewed changes

environments/.stackhpc/inventory/group_vars/all/bastion.yml Outdated Show resolved Hide resolved

bertiethorpe added 4 commits July 25, 2024 10:31

change azimuth collection in bootstrap

e4339bd

Merge branch 'feat/linux-ansible-init' of github.com:stackhpc/ansible…

d5e2c9e

…-slurm-appliance into feat/linux-ansible-init

update azimuth image utils version

627064a

update requirements

d0cdc2a

bertiethorpe previously approved these changes Jul 25, 2024

View reviewed changes

bertiethorpe dismissed their stale review via fb35fb2 July 25, 2024 12:37

bertiethorpe added 2 commits July 25, 2024 13:37

Update bastion.yml

fb35fb2

Merge branch 'main' into feat/linux-ansible-init

24379c0

bertiethorpe added 2 commits July 25, 2024 16:02

Use azimuth image utils collection for ansible-init

1724bf8

Merge branch 'feat/linux-ansible-init' of github.com:stackhpc/ansible…

325c7b4

…-slurm-appliance into feat/linux-ansible-init

bump fatimage

283ded6

bertiethorpe marked this pull request as ready for review July 26, 2024 09:21

bertiethorpe requested a review from a team as a code owner July 26, 2024 09:21

sjpb mentioned this pull request Aug 6, 2024

Support ansible-init in slurm appliance azimuth-cloud/ansible-collection-azimuth-ops#588

Merged

bertiethorpe assigned sjpb and bertiethorpe and unassigned sjpb Aug 6, 2024

mkjpryor approved these changes Aug 7, 2024

View reviewed changes

sjpb merged commit eea9b74 into main Aug 7, 2024
1 check passed

sjpb deleted the feat/linux-ansible-init branch August 7, 2024 09:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Support ansible-init for remote collections #411

Support ansible-init for remote collections #411

Uh oh!

sjpb commented Jul 16, 2024 •

edited by bertiethorpe

Loading

Uh oh!

bertiethorpe commented Jul 18, 2024

Uh oh!

bertiethorpe commented Jul 19, 2024

Uh oh!

bertiethorpe commented Jul 22, 2024

Uh oh!

bertiethorpe commented Jul 22, 2024

Uh oh!

bertiethorpe commented Jul 23, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bertiethorpe commented Jul 25, 2024

Uh oh!

bertiethorpe commented Jul 25, 2024

Uh oh!

sjpb commented Aug 6, 2024 •

edited

Loading

Uh oh!

mkjpryor left a comment

Uh oh!

Uh oh!

Uh oh!

Support ansible-init for remote collections #411

Support ansible-init for remote collections #411

Uh oh!

Conversation

sjpb commented Jul 16, 2024 • edited by bertiethorpe Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bertiethorpe commented Jul 18, 2024

Uh oh!

bertiethorpe commented Jul 19, 2024

Uh oh!

bertiethorpe commented Jul 22, 2024

Uh oh!

bertiethorpe commented Jul 22, 2024

Uh oh!

bertiethorpe commented Jul 23, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bertiethorpe commented Jul 25, 2024

Uh oh!

bertiethorpe commented Jul 25, 2024

Uh oh!

sjpb commented Aug 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mkjpryor left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sjpb commented Jul 16, 2024 •

edited by bertiethorpe

Loading

sjpb commented Aug 6, 2024 •

edited

Loading