Merge pull request #1385 from stackhpc/2024.1-2023.1-merge

2024.1: 2023.1 merge

Author: priteau

.automation

@@ -1,3 +1,5 @@
+.. _stackhpc_release_train:
+
 ======================
 StackHPC Release Train
 ======================

doc/source/configuration/release-train.rst

@@ -63,18 +63,20 @@ The following steps describe the process to test the new package and container r
 Creating the multinode environments
 -----------------------------------
 
-There is a comprehensive guide to setting up a multinode environment with Terraform, found here: https://github.com/stackhpc/terraform-kayobe-multinode. There are some things to note:
+The `Multinode deployment workflow <https://github.com/stackhpc/stackhpc-kayobe-config/actions/workflows/stackhpc-multinode.yml>`_ can be used to automatically test changes.
+
+To manually test the changes, there is a comprehensive guide to set up a Multinode environment with Terraform, found here: https://github.com/stackhpc/terraform-kayobe-multinode. There are some things to note:
 
 * OVN is enabled by default, you should override it under ``etc/kayobe/environments/ci-multinode/kolla.yml kolla_enable_ovn: false`` for the OVS multinode environment.
 
-* Remember to set different vxlan_vnis for each.
+* Remember to set a different ``vxlan_vni`` for each.
 
-* Before starting any tests, run ``dnf distro-sync`` on each host to ensure you are using the same snapshots as in the release train. You can do this using the following commands:
+* Before starting any tests, run ``dnf distro-sync -y`` on each host to ensure you are using the same snapshots as in the release train. Option ``-y`` is used to prevent hosts hang waiting for the confirmation input. You can do this using the following commands:
 
    .. code-block:: console
 
-      kayobe seed host command run -b --command "dnf distro-sync"
-      kayobe overcloud host command run -b --command "dnf distro-sync"
+      kayobe seed host command run -b --command "dnf distro-sync -y"
+      kayobe overcloud host command run -b --command "dnf distro-sync -y"
 
 * This may have installed a new kernel version. If so, you will need to reboot the overcloud hosts. You can check the installed kernels and the currently running kernel with the following commands. If the latest listed version is not running, you will need to reboot.
 
@@ -85,7 +87,7 @@ There is a comprehensive guide to setting up a multinode environment with Terraf
 
    kayobe playbook run --limit seed,overcloud $KAYOBE_CONFIG_PATH/ansible/reboot.yml
 
-* The tempest tests run automatically at the end of deploy-openstack.sh. If you have the time, it is worth fixing any failing tests you can so that there is greater coverage for the package updates. (Also remember to propose these fixes in the relevant repos where applicable.)
+* The tempest tests run automatically at the end of the multinode deployment script. If you have the time, it is worth fixing any failing tests you can so that there is greater coverage for the package updates. (Also remember to propose these fixes in the relevant repos where applicable.)
 
 Upgrading host packages
 -----------------------
@@ -102,6 +104,7 @@ For Rocky Linux 9, bump the snapshot versions in /etc/yum/repos.d with:
 
 .. code-block:: console
 
+   kayobe seed host configure -t dnf
    kayobe overcloud host configure -t dnf
 
 Install new packages:
@@ -112,22 +115,32 @@ Install new packages:
 
 Perform a rolling reboot of hosts:
 
+.. note::
+   In the Multinode environment, the seed-hypervisor cannot access control
+   plane instances with the Openstack client. To use Openstack client, connect
+   to the Seed instance via SSH first. For authentication, use scp to copy
+   ``public-openrc.sh`` to the Seed
+
 .. code-block:: console
 
-   export ANSIBLE_SERIAL=1
-   kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/reboot.yml --limit controllers
-   kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/reboot.yml --limit compute[0]
+   # Check your hypervisor hostname
+   (seed) openstack hypervisor list
+
+   # Reboot controller instances and zeroth compute instance
+   (seed-hypervisor) export ANSIBLE_SERIAL=1
+   (seed-hypervisor) kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/reboot.yml --limit controllers
+   (seed-hypervisor) kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/reboot.yml --limit compute[0]
 
    # Test live migration
-   openstack server create --image cirros --flavor m1.tiny --network external --hypervisor-hostname antelope-pkg-refresh-ovs-compute-02.novalocal --os-compute-api-version 2.74 server1
-   openstack server migrate --live-migration server1
-   watch openstack server show server1
+   (seed) openstack server create --image cirros --flavor m1.tiny --network external --hypervisor-hostname <Your Hypervisor Hostname> --os-compute-api-version 2.74 server1
+   (seed) openstack server migrate --live-migration server1
+   (seed) watch openstack server show server1
 
-   kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/reboot.yml --limit compute[1]
+   (seed-hypervisor) kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/reboot.yml --limit compute[1]
 
    # Try and migrate back
-   openstack server migrate --live-migration server1
-   watch openstack server show server1
+   (seed) openstack server migrate --live-migration server1
+   (seed) watch openstack server show server1
 
 Upgrading containers within a release
 -------------------------------------

doc/source/contributor/package-updates.rst

@@ -449,9 +449,8 @@ To upgrade the Ansible control host:
 Syncing Release Train artifacts
 -------------------------------
 
-New `StackHPC Release Train <../configuration/release-train>` content should be
-synced to the local Pulp server. This includes host packages (Deb/RPM) and
-container images.
+New :ref:`stackhpc_release_train` content should be synced to the local Pulp
+server. This includes host packages (Deb/RPM) and container images.
 
 .. _sync-rt-package-repos:
 
@@ -968,17 +967,27 @@ would be applied:
    kayobe overcloud host configure --check --diff
 
 When ready to apply the changes, it may be advisable to do so in batches, or at
-least start with a small number of hosts.:
+least start with a small number of hosts:
 
 .. code-block:: console
 
    kayobe overcloud host configure --limit <host>
 
-Alternatively, to apply the configuration to all hosts:
 
-.. code-block:: console
+.. warning::
+
+   Take extra care when configuring Ceph hosts. Set the hosts to maintenance
+   mode before reconfiguring them, and unset when done:
+
+   .. code-block:: console
+
+      kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/ceph-enter-maintenance.yml --limit <host>
+      kayobe overcloud host configure --limit <host>
+      kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/ceph-exit-maintenance.yml --limit <host>
 
-   kayobe overcloud host configure
+   **Always** reconfigure hosts in small batches or one-by-one. Check the Ceph
+   state after each host configuration. Ensure all warnings and errors are
+   resolved before moving on.
 
 .. _building_ironic_deployment_images:
 

doc/source/operations/upgrading-openstack.rst

@@ -15,59 +15,61 @@
   tags: os_capacity
   gather_facts: false
   tasks:
-    - name: Create os-capacity directory
-      ansible.builtin.file:
-        path: /opt/kayobe/os-capacity/
-        state: directory
-      when: stackhpc_enable_os_capacity
-
-    - name: Read admin-openrc credential file
-      ansible.builtin.command:
-        cmd: "cat {{ lookup('ansible.builtin.env', 'KOLLA_CONFIG_PATH') }}/admin-openrc.sh"
+    - name: Check if admin-openrc.sh exists
+      ansible.builtin.stat:
+        path: "{{ lookup('ansible.builtin.env', 'KOLLA_CONFIG_PATH') }}/admin-openrc.sh"
       delegate_to: localhost
-      register: credential
-      when: stackhpc_enable_os_capacity
-      changed_when: false
+      register: openrc_file_stat
+      run_once: true
 
-    - name: Set facts for admin credentials
-      ansible.builtin.set_fact:
-        stackhpc_os_capacity_auth_url: "{{ credential.stdout_lines | select('match', '.*OS_AUTH_URL*.') | first | split('=') | last | replace(\"'\",'') }}"
-        stackhpc_os_capacity_project_name: "{{ credential.stdout_lines | select('match', '.*OS_PROJECT_NAME*.') | first | split('=') | last | replace(\"'\",'') }}"
-        stackhpc_os_capacity_domain_name: "{{ credential.stdout_lines | select('match', '.*OS_PROJECT_DOMAIN_NAME*.') | first | split('=') | last | replace(\"'\",'') }}"
-        stackhpc_os_capacity_openstack_region_name: "{{ credential.stdout_lines | select('match', '.*OS_REGION_NAME*.') | first | split('=') | last | replace(\"'\",'') }}"
-        stackhpc_os_capacity_username: "{{ credential.stdout_lines | select('match', '.*OS_USERNAME*.') | first | split('=') | last | replace(\"'\",'') }}"
-        stackhpc_os_capacity_password: "{{ credential.stdout_lines | select('match', '.*OS_PASSWORD*.') | first | split('=') | last | replace(\"'\",'') }}"
-      when: stackhpc_enable_os_capacity
+    - block:
+        - name: Create os-capacity directory
+          ansible.builtin.file:
+            path: /opt/kayobe/os-capacity/
+            state: directory
 
-    - name: Template clouds.yml
-      ansible.builtin.template:
-        src: templates/os_capacity-clouds.yml.j2
-        dest: /opt/kayobe/os-capacity/clouds.yaml
-      when: stackhpc_enable_os_capacity
-      register: clouds_yaml_result
+        - name: Read admin-openrc credential file
+          ansible.builtin.command:
+            cmd: "cat {{ lookup('ansible.builtin.env', 'KOLLA_CONFIG_PATH') }}/admin-openrc.sh"
+          delegate_to: localhost
+          register: credential
+          changed_when: false
 
-    - name: Copy CA certificate to OpenStack Capacity nodes
-      ansible.builtin.copy:
-        src: "{{ stackhpc_os_capacity_openstack_cacert }}"
-        dest: /opt/kayobe/os-capacity/cacert.pem
-      when:
-        - stackhpc_enable_os_capacity
-        - stackhpc_os_capacity_openstack_cacert | length > 0
-      register: cacert_result
+        - name: Set facts for admin credentials
+          ansible.builtin.set_fact:
+            stackhpc_os_capacity_auth_url: "{{ credential.stdout_lines | select('match', '.*OS_AUTH_URL*.') | first | split('=') | last | replace(\"'\",'') }}"
+            stackhpc_os_capacity_project_name: "{{ credential.stdout_lines | select('match', '.*OS_PROJECT_NAME*.') | first | split('=') | last | replace(\"'\",'') }}"
+            stackhpc_os_capacity_domain_name: "{{ credential.stdout_lines | select('match', '.*OS_PROJECT_DOMAIN_NAME*.') | first | split('=') | last | replace(\"'\",'') }}"
+            stackhpc_os_capacity_openstack_region_name: "{{ credential.stdout_lines | select('match', '.*OS_REGION_NAME*.') | first | split('=') | last | replace(\"'\",'') }}"
+            stackhpc_os_capacity_username: "{{ credential.stdout_lines | select('match', '.*OS_USERNAME*.') | first | split('=') | last | replace(\"'\",'') }}"
+            stackhpc_os_capacity_password: "{{ credential.stdout_lines | select('match', '.*OS_PASSWORD*.') | first | split('=') | last | replace(\"'\",'') }}"
 
-    - name: Ensure os_capacity container is running
-      community.docker.docker_container:
-        name: os_capacity
-        image: ghcr.io/stackhpc/os-capacity:master
-        env:
-          OS_CLOUD: openstack
-          OS_CLIENT_CONFIG_FILE: /etc/openstack/clouds.yaml
-        mounts:
-          - type: bind
-            source: /opt/kayobe/os-capacity/
-            target: /etc/openstack/
-        network_mode: host
-        restart: "{{ clouds_yaml_result is changed or cacert_result is changed }}"
-        restart_policy: unless-stopped
-      become: true
-      when: stackhpc_enable_os_capacity
+        - name: Template clouds.yml
+          ansible.builtin.template:
+            src: templates/os_capacity-clouds.yml.j2
+            dest: /opt/kayobe/os-capacity/clouds.yaml
+          register: clouds_yaml_result
+
+        - name: Copy CA certificate to OpenStack Capacity nodes
+          ansible.builtin.copy:
+            src: "{{ stackhpc_os_capacity_openstack_cacert }}"
+            dest: /opt/kayobe/os-capacity/cacert.pem
+          when: stackhpc_os_capacity_openstack_cacert | length > 0
+          register: cacert_result
+
+        - name: Ensure os_capacity container is running
+          community.docker.docker_container:
+            name: os_capacity
+            image: ghcr.io/stackhpc/os-capacity:{{ stackhpc_os_capacity_version }}
+            env:
+              OS_CLOUD: openstack
+              OS_CLIENT_CONFIG_FILE: /etc/openstack/clouds.yaml
+            mounts:
+              - type: bind
+                source: /opt/kayobe/os-capacity/
+                target: /etc/openstack/
+            network_mode: host
+            restart: "{{ clouds_yaml_result is changed or cacert_result is changed }}"
+            restart_policy: unless-stopped
+          become: true
+      when: stackhpc_enable_os_capacity and openrc_file_stat.stat.exists

etc/kayobe/ansible/deploy-os-capacity-exporter.yml

@@ -51,6 +51,9 @@ rhel9cis_rule_6_1_15: false
 # filesystem.  We do not want to change /var/lib/docker permissions.
 rhel9cis_no_world_write_adjust: false
 
+# Prevent hardening from recursivley changing permissions on log files
+rhel9cis_rule_4_2_3: false
+
 # Configure log rotation to prevent audit logs from filling the disk
 rhel9cis_auditd:
   space_left_action: syslog
@@ -153,6 +156,9 @@ ubtu22cis_no_owner_adjust: false
 ubtu22cis_no_world_write_adjust: false
 ubtu22cis_suid_adjust: false
 
+# Prevent hardening from recursivley changing permissions on log files
+ubtu22cis_rule_4_2_3: false
+
 # Configure log rotation to prevent audit logs from filling the disk
 ubtu22cis_auditd:
   action_mail_acct: root

etc/kayobe/inventory/group_vars/cis-hardening/cis

@@ -30,7 +30,9 @@
 
 # List of additional Diskimage Builder (DIB) elements to use when building IPA
 # images. Default is none.
-#ipa_build_dib_elements_extra:
+ipa_build_dib_elements_extra:
+  - extra-hardware
+  - mellanox
 
 # List of Diskimage Builder (DIB) elements to use when building IPA images.
 # Default is combination of ipa_build_dib_elements_default and
@@ -117,7 +119,10 @@
 #ipa_collectors_default:
 
 # List of additional inspection collectors to run.
-#ipa_collectors_extra:
+ipa_collectors_extra:
+  - "dmi-decode"
+  - "extra-hardware"
+  - "numa-topology"
 
 # List of inspection collectors to run.
 #ipa_collectors:
@@ -135,7 +140,11 @@
 #ipa_kernel_options_default:
 
 # List of additional kernel parameters for Ironic python agent.
-#ipa_kernel_options_extra:
+ipa_kernel_options_extra:
+  # Useful until NTP is configured by default
+  - ipa-insecure=1
+  # Avoid disk benchmark failures on some NVMe drives
+  - nvme_core.multipath=N
 
 # List of kernel parameters for Ironic python agent.
 #ipa_kernel_options:

etc/kayobe/ipa.yml

@@ -6,6 +6,9 @@ kolla_image_tags:
   openstack:
     rocky-9: 2024.1-rocky-9-20240903T113235
     ubuntu-jammy: 2024.1-ubuntu-jammy-20240917T091559
+  blazar:
+    rocky-9: 2024.1-rocky-9-20241125T093138
+    ubuntu-jammy: 2024.1-ubuntu-jammy-20241125T093138
   heat:
     rocky-9: 2024.1-rocky-9-20240805T142526
   nova:

etc/kayobe/kolla-image-tags.yml

@@ -150,6 +150,10 @@ kolla_sources:
     type: git
     location: https://github.com/stackhpc/octavia.git
     reference: stackhpc/{{ openstack_release }}
+  blazar-base:
+    type: git
+    location: https://github.com/stackhpc/blazar
+    reference: stackhpc/master
 
 ###############################################################################
 # Kolla image build configuration.