Promote multiple tags simultaneously based on Kayobe config

This modifies the container image promotion process to be similar to
package repo promotion. The tags for each image are stored in files in
StackHPC Kayobe configuration (one per base distro), and these are
queried to determine which tag to promote for each image.

This avoids the need to specify tags when promoting images, and will
allow for automatic tag promotion after merges to StackHPC Kayobe
Configuration.

Author: markgoddard

.github/workflows/container-promote.yml

@@ -13,9 +13,10 @@ on:
         type: string
         required: false
         default: ""
-      tag:
-        description: Container image tag to promote
-        required: true
+      kayobe_config_branch:
+        required: false
+        description: Branch of StackHPC Kayobe configuration to use
+        default: stackhpc/yoga
 
 env:
   ANSIBLE_FORCE_COLOR: True
@@ -43,15 +44,22 @@ jobs:
         pip install -r requirements.txt &&
         ansible-galaxy collection install -r requirements.yml -p ansible/collections
 
+    - name: Clone StackHPC Kayobe configuration repository
+      uses: actions/checkout@v3
+      with:
+        repository: stackhpc/stackhpc-kayobe-config
+        ref: refs/heads/${{ github.event.inputs.kayobe_config_branch }}
+        path: stackhpc-kayobe-config
+
     - name: Promote images from stackhpc-dev to stackhpc namespace in Ark
       run: |
         source venv/bin/activate &&
         ansible-playbook -i ansible/inventory \
+        ansible/dev-pulp-container-tag-query-kayobe.yml \
         ansible/dev-pulp-container-promote.yml \
-        -e dev_pulp_repository_container_promotion_tag="$TAG" \
         -e kolla_container_image_filter="'$FILTER'" \
-        -e kolla_base_distros_override="'$DISTROS'"
+        -e kolla_base_distros_override="'$DISTROS'" \
+        -e kayobe_config_repo_path=./stackhpc-kayobe-config/
       env:
-        TAG: ${{ github.event.inputs.tag }}
         FILTER: ${{ github.event.inputs.filter }}
         DISTROS: ${{ github.event.inputs.distros }}

ansible/dev-pulp-container-promote.yml

@@ -1,22 +1,28 @@
 ---
 # This playbook promotes images in the development namespace (stackhpc-dev) to
 # the release namespace (stackhpc). This makes them available to clients.
-# Images with a tag defined by dev_pulp_repository_container_promotion_tag are
-# promoted.
+# Images with a tag defined by dev_pulp_repository_container_promotion_tags are
+# promoted. dev_pulp_repository_container_promotion_tags is a dict with the
+# following format:
+#
+# dev_pulp_repository_container_promotion_tags:
+#   <base distro>:
+#     <image name>: <image tag>
 
 - name: Promote dev Pulp containers
   hosts: localhost
   gather_facts: false
   tasks:
-    - name: Fail if container image to promote is not defined
+    - name: Fail if container images to promote are not defined
       fail:
         msg: >
-          The container image to promote must be specified via
-          'dev_pulp_repository_container_promotion_tag'
-      when: dev_pulp_repository_container_promotion_tag is not defined
+          The container images to promote must be specified via
+          'dev_pulp_repository_container_promotion_tags'
+      when: dev_pulp_repository_container_promotion_tags is not defined
 
-    - debug:
-        msg: "Promoting tag {{ dev_pulp_repository_container_promotion_tag }}"
+    - name: Display which tags are being promoted
+      debug:
+        var: "dev_pulp_repository_container_promotion_tags"
 
     # Copy tags from stackhpc-dev to stackhpc repositories.
     - import_role:
@@ -27,20 +33,18 @@
         pulp_password: "{{ dev_pulp_password }}"
         pulp_container_content: >-
           {%- set contents = [] -%}
-          {%- for base_distro in kolla_base_distros -%}
-          {%- for image in kolla_container_images_filtered -%}
-          {%- if image not in kolla_unbuildable_images[base_distro]-%}
+          {%- for base_distro, distro_images in dev_pulp_repository_container_promotion_tags.items() -%}
+          {%- for image, tag in distro_images.items() -%}
           {%- set src_image_repo = "stackhpc-dev/" ~ base_distro ~ "-source-" ~ image -%}
           {%- set dest_image_repo = "stackhpc/" ~ base_distro ~ "-source-" ~ image -%}
           {%- set content = {
-            "allow_missing": True,
+            "allow_missing": False,
             "src_repo": src_image_repo,
             "src_is_push": true,
             "repository": dest_image_repo,
-            "tags": [dev_pulp_repository_container_promotion_tag],
+            "tags": [tag],
           } -%}
           {%- set _ = contents.append(content) -%}
-          {%- endif -%}
           {%- endfor -%}
           {%- endfor -%}
           {{ contents }}

ansible/dev-pulp-container-tag-query-kayobe.yml

@@ -0,0 +1,41 @@
+---
+# This playbook queries the Pulp container image tags defined in a Kayobe
+# configuration repository (in etc/kayobe/kolla/tags/*.yml). It then sets the
+# 'dev_pulp_repository_container_promotion_tags' tag map variable based
+# upon those tags, which defines the set of container image tags that
+# will be promoted when the dev-pulp-container-promote.yml playbook is run.
+
+- import_playbook: kayobe-container-tag-query.yml
+
+- name: Set dev Pulp container image tags to promote from Kayobe config tags
+  hosts: localhost
+  gather_facts: true
+  tasks:
+    - name: Set a fact about container image tags to promote
+      set_fact:
+        dev_pulp_repository_container_promotion_tags: >-
+          {%- set tags = {} -%}
+          {%- for base_distro in kolla_base_distros -%}
+          {%- for image in kolla_container_images_filtered -%}
+          {%- if image not in kolla_unbuildable_images[base_distro] -%}
+          {%- if image in kayobe_pulp_container_tags.get(base_distro, {}) -%}
+          {%- set distro_images = tags.setdefault(base_distro, {}) -%}
+          {%- set _ = distro_images.setdefault(image, kayobe_pulp_container_tags[base_distro][image]) -%}
+          {%- endif -%}
+          {%- endif -%}
+          {%- endfor -%}
+          {%- endfor -%}
+          {{ tags }}
+
+    - name: Display container promotion tags facts
+      debug:
+        var: dev_pulp_repository_container_promotion_tags
+
+    - name: Assert that tags variable is populated
+      assert:
+        that:
+          - >-
+            dev_pulp_repository_container_promotion_tags | length > 0
+        msg: >-
+          Distribution promotion tag variable
+          'dev_pulp_repository_container_promotion_tags' is empty

ansible/kayobe-container-tag-query.yml

@@ -0,0 +1,58 @@
+---
+# This playbook queries the Pulp container image tags defined in a Kayobe
+# configuration repository (in etc/kayobe/kolla/tags/*.yml). It then sets
+# the 'kayobe_pulp_container_tags' tag map variable based upon those
+# tags.
+#
+# The kayobe-config repository path should be specified via
+# kayobe_config_repo_path.
+
+- name: Query container image tags for Kayobe
+  hosts: localhost
+  gather_facts: True
+  vars:
+    kayobe_config_repo_path: ""
+    kayobe_pulp_container_tags_path: "{{ (kayobe_config_repo_path ~ '/etc/kayobe/kolla/tags') | realpath }}"
+  tasks:
+    - name: Fail if Kayobe config repo path is not specified
+      fail:
+        msg: >
+          Kayobe config git repository path must be specified via 'kayobe_config_repo_path'.
+      when: not kayobe_config_repo_path
+
+    - name: Fail if Kayobe config repo path is not a directory
+      fail:
+        msg: >
+          Kayobe config git repository path {{ kayobe_config_repo_path }} is not a directory.
+      when: kayobe_config_repo_path is not directory
+
+    - name: Fail if Kayobe Pulp container image tags directory not found
+      fail:
+        msg: >
+          Can't find Kayobe Pulp container image tags directory {{ kayobe_pulp_container_tags_path }}.
+      when: kayobe_pulp_container_tags_path is not directory
+
+    - name: Find container image tag files
+      find:
+        paths: "{{ kayobe_pulp_container_tags_path }}"
+        patterns: "*.yml"
+      register: find_result
+
+    - name: Assert that a container image tag file was found
+      assert:
+        that:
+          find_result.matched > 0
+
+    - name: Set a fact about kayobe Pulp container image tags
+      vars:
+        base_distro: "{{ item.path | basename | splitext | first }}"
+        distro_tags: "{{ lookup('file', item.path) | from_yaml }}"
+      set_fact:
+        kayobe_pulp_container_tags: "{{ kayobe_pulp_container_tags | default({}) | combine({base_distro: distro_tags}) }}"
+      loop: "{{ find_result.files }}"
+      loop_control:
+        label: "{{ item.path }}"
+
+    - name: Display Kayobe Pulp container image tags
+      debug:
+        var: kayobe_pulp_container_tags

docs/usage/content-workflows.md

@@ -245,17 +245,22 @@ skydive_analyzer_tag: wallaby-20220811T091848
 
 The [Promote container repositories](https://github.com/stackhpc/stackhpc-release-train/actions/workflows/container-promote.yml) workflow runs on demand.
 It should be run when container images need to be released, typically after a change to [update container image tags](#updating-container-image-tags-in-kayobe-configuration) has been approved.
-It runs the following playbook:
+It runs the following playbooks:
 
-* `dev-pulp-container-promote.yml`: Promote a set of container images from `stackhpc-dev` to `stackhpc` namespace. The tag to be promoted is defined via `dev_pulp_repository_container_promotion_tag` which should be specified as an extra variable (`-e`).
+* `dev-pulp-container-tag-query-kayobe.yml`: Query the Pulp container image tags defined in a Kayobe configuration repository and set the tag map variable `dev_pulp_repository_container_promotion_tags` based upon those tags. A path to a Kayobe configuration repository must be specified via `kayobe_config_repo_path`.
+* `dev-pulp-container-promote.yml`: Promote a set of container images from `stackhpc-dev` to `stackhpc` namespace. The tags to be promoted are defined via `dev_pulp_repository_container_promotion_tags`.
 
 Use GitHub Actions to run this workflow, or to run it manually:
 
 ```
 ansible-playbook -i ansible/inventory \
-ansible/dev-pulp-container-promote.yml
+ansible/dev-pulp-container-tag-query-kayobe.yml \
+ansible/dev-pulp-container-promote.yml \
+-e kayobe_config_repo_path=../stackhpc-kayobe-config/
 ```
 
+In this example, the Pulp container image tags defined in the `etc/kayobe/kolla/tags/<distro>.yml` files in `../stackhpc-kayobe-config` repository (relative to the current working directory) will be promoted to releases.
+
 ## Other utilities
 
 * `dev-pulp-distribution-list.yml`: List available distributions in `ark`.