Merge remote-tracking branch 'origin/stackhpc/2023.1' into sync-antelope-caracal

Author: Alex-Welsh

doc/source/_static/images/release-train.svg

@@ -347,6 +347,10 @@ should be used in the Kolla Manila configuration e.g.:
 RADOS Gateways
 --------------
 
+RADOS Gateway integration is described in the :kolla-ansible-doc:`Kolla Ansible
+documentation
+<https://docs.openstack.org/kolla-ansible/latest/reference/storage/external-ceph-guide.html#radosgw>`.
+
 RADOS Gateways (RGWs) are defined with the following:
 
 .. code:: yaml
@@ -377,7 +381,7 @@ The set of commands below configure all of these.
   - "config set client.rgw rgw_enable_apis 's3, swift, swift_auth, admin'"
   - "config set client.rgw rgw_enforce_swift_acls true"
   - "config set client.rgw rgw_keystone_accepted_admin_roles 'admin'"
-  - "config set client.rgw rgw_keystone_accepted_roles 'member, Member, _member_, admin'"
+  - "config set client.rgw rgw_keystone_accepted_roles 'member, admin'"
   - "config set client.rgw rgw_keystone_admin_domain Default"
   - "config set client.rgw rgw_keystone_admin_password {{ secrets_ceph_rgw_keystone_password }}"
   - "config set client.rgw rgw_keystone_admin_project service"
@@ -393,6 +397,12 @@ The set of commands below configure all of these.
   - "config set client.rgw rgw_swift_account_in_url true"
   - "config set client.rgw rgw_swift_versioning_enabled true"
 
+Enable the Kolla Ansible RADOS Gateway integration in ``kolla.yml``:
+
+.. code:: yaml
+
+   kolla_enable_ceph_rgw: true
+
 As we have configured Ceph to respond to Swift APIs, you will need to tell
 Kolla to account for this when registering Swift endpoints with Keystone. Also,
 when ``rgw_swift_account_in_url`` is set, the equivalent Kolla variable should
@@ -414,6 +424,11 @@ before deploying the RADOS gateways. If you are using the Kolla load balancer
 
   kayobe overcloud service deploy -kt ceph-rgw,keystone,haproxy,loadbalancer
 
+There are two options for load balancing RADOS Gateway:
+
+1. HA with Ceph Ingress services
+2. RGWs with hyper-converged Ceph (using the Kolla Ansible deployed HAProxy
+   load balancer)
 
 .. _RGWs-with-hyper-converged-Ceph:
 

doc/source/configuration/cephadm.rst

@@ -60,12 +60,12 @@ To deploy the CAPI management cluster using this site-specific environment, run
 
 .. code-block:: bash
 
-    # Activate the environment
-    ./bin/activate <site-specific-name>
-
     # Install or update the local Ansible Python venv
     ./bin/ensure-venv
 
+    # Activate the environment
+    source bin/activate <site-specific-name>
+
     # Install or update Ansible dependencies
     ansible-galaxy install -f -r ./requirements.yml
 
@@ -103,12 +103,7 @@ To configure the Magnum service with the Cluster API driver enabled, first ensur
 
 Next, copy the CAPI management cluster's kubeconfig file into your stackhpc-kayobe-config environment (e.g. ``<your-skc-environment>/kolla/config/magnum/kubeconfig``). This file must be Ansible vault encrypted.
 
-The following config should also be set in your stackhpc-kayobe-config environment:
-
-.. code-block:: yaml
-    :caption: kolla/globals.yml
-
-    magnum_capi_helm_driver_enabled: true
+The presence of a kubeconfig file in the Magnum config directory is used by Kolla to determine whether the CAPI Helm driver should be enabled.
 
 To apply the configuration, run ``kayobe overcloud service reconfigure -kt magnum``.
 

doc/source/configuration/magnum-capi.rst

@@ -12,7 +12,6 @@ The short version
    particular the defaults assume that the ``provision_oc_net`` network will be
    used.
 #. Generate secrets: ``kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/wazuh-secrets.yml``
-#. Encrypt the secrets: ``ansible-vault encrypt --vault-password-file ~/vault.password  $KAYOBE_CONFIG_PATH/environments/ci-multinode/wazuh-secrets.yml``
 #. Deploy the Wazuh manager: ``kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/wazuh-manager.yml``
 #. Deploy the Wazuh agents: ``kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/wazuh-agent.yml``
 
@@ -250,7 +249,6 @@ It will be used by wazuh secrets playbook to generate wazuh secrets vault file.
 .. code-block:: console
 
   kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/wazuh-secrets.yml
-  ansible-vault encrypt --vault-password-file ~/vault.pass $KAYOBE_CONFIG_PATH/wazuh-secrets.yml
 
 Configure Wazuh Dashboard's Server Host
 ---------------------------------------

doc/source/configuration/wazuh.rst

@@ -9,7 +9,7 @@ collections:
   - name: stackhpc.pulp
     version: 0.5.5
   - name: stackhpc.hashicorp
-    version: 2.5.0
+    version: 2.5.1
   - name: stackhpc.kayobe_workflows
     version: 1.0.3
 roles:

etc/kayobe/ansible/requirements.yml

@@ -142,4 +142,4 @@ if ! $KOLLA_OPENSTACK_COMMAND flavor list | grep -q m1.tiny; then
     $KOLLA_OPENSTACK_COMMAND flavor create --id 5 --ram 16384 --disk 160 --vcpus 8 m1.xlarge
 fi
 
-touch /tmp/.init-runonce
+touch /tmp/.init-runonce

etc/kayobe/ansible/scripts/aio-init.sh

@@ -74,15 +74,15 @@ stackhpc_apt_repositories:
 # Do not replace apt configuration for non-overcloud hosts. This can result in
 # errors if apt reconfiguration is performed before local repository mirrors
 # are deployed.
-apt_repositories: "{{ stackhpc_apt_repositories | selectattr('required') | list if 'overcloud' in group_names else [] }}"
+apt_repositories: "{{ stackhpc_apt_repositories | selectattr('required') | list if stackhpc_repos_enabled | bool else [] }}"
 
 # Whether to disable repositories in /etc/apt/sources.list. This may be used
 # when replacing the distribution repositories via apt_repositories.
 # Default is false.
 # Do not disable the default apt configuration for non-overcloud hosts. This
 # can result in errors if apt reconfiguration is performed before local
 # repository mirrors are deployed.
-apt_disable_sources_list: "{{ 'overcloud' in group_names }}"
+apt_disable_sources_list: "{{ stackhpc_repos_enabled | bool }}"
 
 # Apt auth configuration for accessing the package repository mirror.
 stackhpc_apt_auth:
@@ -98,7 +98,7 @@ stackhpc_apt_auth:
 # * filename: Name of a file in which to store the auth configuration. The
 #   extension should be '.conf'.
 # Default is an empty list.
-apt_auth: "{{ stackhpc_apt_auth if 'overcloud' in group_names and stackhpc_repo_mirror_username is truthy else [] }}"
+apt_auth: "{{ stackhpc_apt_auth if stackhpc_repos_enabled | bool and stackhpc_repo_mirror_username is truthy else [] }}"
 
 ###############################################################################
 # Dummy variable to allow Ansible to accept this file.

etc/kayobe/apt.yml

@@ -41,10 +41,10 @@
 #     file: myrepo
 #     gpgkey: http://gpgkey
 #     gpgcheck: yes
-#dnf_custom_repos:
+dnf_custom_repos: "{{ stackhpc_dnf_repos if stackhpc_repos_enabled | bool else [] }}"
 
 # A dict of custom repositories that point to the local Pulp server.
-# To use these repos, set dnf_custom_repos to the value of stackhpc_dnf_repos.
+# To use these repos, set stackhpc_repos_enabled to true.
 # This is done by default for hosts in the overcloud group via a group_vars
 # file.
 stackhpc_dnf_repos: "{{ dnf_custom_repos_el9 | combine(dnf_custom_repos_rocky_9) | combine(dnf_custom_repos_elrepo_9 if dnf_install_elrepo_9 | bool else {}) }}"

etc/kayobe/dnf.yml

@@ -25,4 +25,4 @@ ansible-galaxy collection install -p ansible/collections -r requirements.yml
 source $BASE_PATH/src/kayobe-config/etc/kolla/public-openrc.sh
 
 # Run script to configure openstack cloud
-tools/openstack-config 
+tools/openstack-config

etc/kayobe/environments/aufn-ceph/configure-openstack.sh

@@ -26,4 +26,4 @@ storage-ceph
 # Monitoring groups
 
 [monitoring:children]
-controllers
+controllers

etc/kayobe/environments/aufn-ceph/inventory/groups

@@ -107,10 +107,11 @@ storage_allocation_pool_end: 192.168.41.254
 storage_vlan: 105
 
 # Storage management network
-storage_mgmt_cidr: 192.168.42.0/24
+# NOTE: Skipping the .42 subnet to avoid a collision with a popular number.
+storage_mgmt_cidr: 192.168.43.0/24
 storage_mgmt_mtu: "{{ ansible_facts.default_ipv4.mtu - 50 }}"
-storage_mgmt_allocation_pool_start: 192.168.42.3
-storage_mgmt_allocation_pool_end: 192.168.42.254
+storage_mgmt_allocation_pool_start: 192.168.43.3
+storage_mgmt_allocation_pool_end: 192.168.43.254
 storage_mgmt_vlan: 106
 
 # Provision overcloud network

etc/kayobe/environments/ci-multinode/networks.yml

@@ -3,4 +3,4 @@
 # Feature flags
 
 # Whether or not to run CIS benchmark hardening playbooks. Default is false.
-stackhpc_enable_cis_benchmark_hardening_hook: false
+stackhpc_enable_cis_benchmark_hardening_hook: false

etc/kayobe/inventory/group_vars/all/stackhpc

@@ -0,0 +1,6 @@
+---
+# Use upstream package repos by default to avoid situations where the
+# seed-hypervisor tries to use a local pulp repo on the seed VM before the seed
+# vm has been provisioned
+# This behaviour is overriden for Overcloud hosts.
+stackhpc_repos_enabled: false

etc/kayobe/inventory/group_vars/all/stackhpc-repos

@@ -1,7 +1,7 @@
+---
 # Only use local pulp mirrors for overcloud hosts
 # to avoid situations where the seed-hypervisor 
 # tries to use a local pulp repo on the seed VM
 # before the seed vm has been provisioned
-dnf_custom_repos: "{{ stackhpc_dnf_repos }}"
-
+stackhpc_repos_enabled: true
 enable_docker_repo: false

etc/kayobe/inventory/group_vars/overcloud/stackhpc-repos

@@ -147,6 +147,14 @@ kolla_sources:
     type: git
     location: https://github.com/stackhpc/networking-generic-switch.git
     reference: stackhpc/{{ openstack_release }}
+  octavia-api-plugin-ovn-octavia-provider:
+    type: git
+    location: https://github.com/stackhpc/ovn-octavia-provider.git
+    reference: stackhpc/{{ openstack_release }}
+  octavia-driver-agent-plugin-ovn-octavia-provider:
+    type: git
+    location: https://github.com/stackhpc/ovn-octavia-provider.git
+    reference: stackhpc/{{ openstack_release }}
 
 ###############################################################################
 # Kolla image build configuration.
@@ -433,16 +441,44 @@ kolla_build_customizations_common:
     - /additions/*
 
 kolla_build_customizations_rocky:
+  kolla_toolbox_packages_remove:
+    - openvswitch
+  kolla_toolbox_packages_append:
+    - openvswitch3.3
+  manila_base_packages_remove:
+    - openvswitch
+  manila_base_packages_append:
+    - openvswitch3.3
+  neutron_base_packages_remove:
+    - openvswitch
+    - python3-openvswitch
+  neutron_base_packages_append:
+    - openvswitch3.3
+    - python3-openvswitch3.3
+  nova_base_packages_remove:
+    - openvswitch
+    - python3-openvswitch
+  nova_base_packages_append:
+    - openvswitch3.3
+    - python3-openvswitch3.3
+  octavia_base_packages_remove:
+    - python3-openvswitch
+  openvswitch_base_packages_remove:
+    - openvswitch
+    - python3-openvswitch
+  openvswitch_base_packages_append:
+    - openvswitch3.3
+    - python3-openvswitch3.3
   ovn_base_packages_override:
-    - ovn23.03
+    - ovn24.03
   ovn_controller_packages_override:
-    - ovn23.03-host
+    - ovn24.03-host
   ovn_nb_db_server_packages_override:
-    - ovn23.03-central
+    - ovn24.03-central
   ovn_northd_packages_override:
-    - ovn23.03-central
+    - ovn24.03-central
   ovn_sb_db_server_packages_override:
-    - ovn23.03-central
+    - ovn24.03-central
 
 kolla_build_customizations_ubuntu: {}
 

etc/kayobe/kolla.yml

@@ -902,4 +902,4 @@
   "version": 1,
   "weekStart": ""
 }
-{% endraw %}
+{% endraw %}

etc/kayobe/kolla/config/grafana/dashboards/openstack/grafana_cloud_dashboard.json

@@ -1138,4 +1138,4 @@
   "version": 1,
   "weekStart": ""
 }
-{% endraw %}
+{% endraw %}

etc/kayobe/kolla/config/grafana/dashboards/openstack/grafana_project_dashboard.json

@@ -2531,4 +2531,4 @@
   "version": 2,
   "weekStart": ""
 }
-{% endraw %}
+{% endraw %}

etc/kayobe/kolla/config/grafana/dashboards/openstack/openstack.json

@@ -5353,4 +5353,4 @@
   "version": 8,
   "weekStart": ""
 }
-{% endraw %}
+{% endraw %}

etc/kayobe/kolla/config/grafana/dashboards/openstack/prometheus_benchmark.json

@@ -44,4 +44,4 @@ receivers:
 
 
 templates:
-  - '/etc/prometheus/*.tmpl'
+  - '/etc/prometheus/*.tmpl'

etc/kayobe/kolla/config/prometheus/prometheus-alertmanager.yml.example

@@ -104,7 +104,7 @@ prometheus_blackbox_exporter_endpoints_default:
       - "{{ ('swift_internal:os_endpoint:' + swift_internal_base_endpoint) if not kolla_same_external_internal_vip | bool }}"
     enabled: "{{ enable_swift | bool }}"
   # Additional service endpoints
-  - endpoints: "{% set etcd_endpoints = [] %}{% for host in groups.get('etcd', []) %}{{ etcd_endpoints.append('etcd_' + host + ':http_2xx:' + hostvars[host]['etcd_protocol'] + '://' + ('api' | kolla_address(host) | put_address_in_context('url')) + ':' + hostvars[host]['etcd_client_port'] + '/metrics')}}{% endfor %}{{ etcd_endpoints }}"
+  - endpoints: "{% set etcd_endpoints = [] %}{% for host in groups.get('etcd', []) %}{{ etcd_endpoints.append('etcd_' + host.replace('-', '') + ':http_2xx:' + hostvars[host]['etcd_protocol'] + '://' + ('api' | kolla_address(host) | put_address_in_context('url')) + ':' + hostvars[host]['etcd_client_port'] + '/metrics')}}{% endfor %}{{ etcd_endpoints }}"
     enabled: "{{ enable_etcd | bool }}"
   - endpoints:
       - "grafana:http_2xx:{{ grafana_public_endpoint }}"
@@ -125,9 +125,9 @@ prometheus_blackbox_exporter_endpoints_default:
   - endpoints:
       - "prometheus_alertmanager:http_2xx_alertmanager:{{ prometheus_alertmanager_public_endpoint if enable_prometheus_alertmanager_external else prometheus_alertmanager_internal_endpoint }}"
     enabled: "{{ enable_prometheus_alertmanager | bool }}"
-  - endpoints: "{% set rabbitmq_endpoints = [] %}{% for host in groups.get('rabbitmq', []) %}{{ rabbitmq_endpoints.append('rabbitmq_' + host + (':tls_connect:' if rabbitmq_enable_tls else ':tcp_connect:') + ('api' | kolla_address(host) | put_address_in_context('url')) + ':' + hostvars[host]['rabbitmq_port'] ) }}{% endfor %}{{ rabbitmq_endpoints }}"
+  - endpoints: "{% set rabbitmq_endpoints = [] %}{% for host in groups.get('rabbitmq', []) %}{{ rabbitmq_endpoints.append('rabbitmq_' + host.replace('-', '') + (':tls_connect:' if rabbitmq_enable_tls else ':tcp_connect:') + ('api' | kolla_address(host) | put_address_in_context('url')) + ':' + hostvars[host]['rabbitmq_port'] ) }}{% endfor %}{{ rabbitmq_endpoints }}"
     enabled: "{{ enable_rabbitmq | bool }}"
-  - endpoints: "{% set redis_endpoints = [] %}{% for host in groups.get('redis', []) %}{{ redis_endpoints.append('redis_' + host + ':tcp_connect:' + ('api' | kolla_address(host) | put_address_in_context('url')) + ':' + hostvars[host]['redis_port']) }}{% endfor %}{{ redis_endpoints }}"
+  - endpoints: "{% set redis_endpoints = [] %}{% for host in groups.get('redis', []) %}{{ redis_endpoints.append('redis_' + host.replace('-', '') + ':tcp_connect:' + ('api' | kolla_address(host) | put_address_in_context('url')) + ':' + hostvars[host]['redis_port']) }}{% endfor %}{{ redis_endpoints }}"
     enabled: "{{ enable_redis | bool }}"
 
 # Ensure service endpoints are defined
@@ -141,10 +141,12 @@ cloudkitty_internal_endpoint: "{{ cloudkitty_internal_fqdn | kolla_url(internal_
 cloudkitty_public_endpoint: "{{ cloudkitty_external_fqdn | kolla_url(public_protocol, cloudkitty_api_public_port) }}"
 gnocchi_internal_endpoint: "{{ gnocchi_internal_fqdn | kolla_url(internal_protocol, gnocchi_api_port) }}"
 gnocchi_public_endpoint: "{{ gnocchi_external_fqdn | kolla_url(public_protocol, gnocchi_api_public_port) }}"
-heat_internal_base_endpoint: "{{ heat_internal_fqdn | kolla_url(internal_protocol, heat_api_port) }}"
-heat_public_base_endpoint: "{{ heat_external_fqdn | kolla_url(public_protocol, heat_api_public_port) }}"
+grafana_public_endpoint: "{{ grafana_external_fqdn | kolla_url(public_protocol, grafana_server_public_port) }}"
 heat_cfn_internal_base_endpoint: "{{ heat_cfn_internal_fqdn | kolla_url(internal_protocol, heat_api_cfn_port) }}"
 heat_cfn_public_base_endpoint: "{{ heat_cfn_external_fqdn | kolla_url(public_protocol, heat_api_cfn_public_port) }}"
+heat_internal_base_endpoint: "{{ heat_internal_fqdn | kolla_url(internal_protocol, heat_api_port) }}"
+heat_public_base_endpoint: "{{ heat_external_fqdn | kolla_url(public_protocol, heat_api_public_port) }}"
+horizon_public_endpoint: "{{ horizon_external_fqdn | kolla_url(public_protocol, horizon_listen_port) }}"
 ironic_inspector_internal_endpoint: "{{ ironic_inspector_internal_fqdn | kolla_url(internal_protocol, ironic_inspector_port) }}"
 ironic_inspector_public_endpoint: "{{ ironic_inspector_external_fqdn | kolla_url(public_protocol, ironic_inspector_public_port) }}"
 magnum_internal_base_endpoint: "{{ magnum_internal_fqdn | kolla_url(internal_protocol, magnum_api_port) }}"
@@ -153,12 +155,12 @@ manila_internal_base_endpoint: "{{ manila_internal_fqdn | kolla_url(internal_pro
 manila_public_base_endpoint: "{{ manila_external_fqdn | kolla_url(public_protocol, manila_api_public_port) }}"
 nova_internal_base_endpoint: "{{ nova_internal_fqdn | kolla_url(internal_protocol, nova_api_port) }}"
 nova_public_base_endpoint: "{{ nova_external_fqdn | kolla_url(public_protocol, nova_api_public_port) }}"
+opensearch_dashboards_external_endpoint: "{{ opensearch_dashboards_external_fqdn | default(kolla_external_fqdn) | kolla_url(public_protocol, opensearch_dashboards_port_external) }}"
+opensearch_dashboards_internal_endpoint: "{{ opensearch_dashboards_internal_fqdn | default(kolla_internal_fqdn) | kolla_url(internal_protocol, opensearch_dashboards_port) }}"
 placement_internal_endpoint: "{{ placement_internal_fqdn | kolla_url(internal_protocol, placement_api_port) }}"
 placement_public_endpoint: "{{ placement_external_fqdn | kolla_url(public_protocol, placement_api_public_port) }}"
-swift_public_base_endpoint: "{{ swift_external_fqdn | kolla_url(public_protocol, swift_proxy_server_port) }}"
-opensearch_dashboards_internal_endpoint: "{{ opensearch_dashboards_internal_fqdn | default(kolla_internal_fqdn) | kolla_url(internal_protocol, opensearch_dashboards_port) }}"
-opensearch_dashboards_external_endpoint: "{{ opensearch_dashboards_external_fqdn | default(kolla_external_fqdn) | kolla_url(public_protocol, opensearch_dashboards_port_external) }}"
-prometheus_internal_endpoint: "{{ prometheus_internal_fqdn | kolla_url(internal_protocol, prometheus_port) }}"
-prometheus_public_endpoint: "{{ prometheus_external_fqdn | kolla_url(public_protocol, prometheus_public_port) }}"
 prometheus_alertmanager_internal_endpoint: "{{ prometheus_alertmanager_internal_fqdn | kolla_url(internal_protocol, prometheus_alertmanager_port) }}"
 prometheus_alertmanager_public_endpoint: "{{ prometheus_alertmanager_external_fqdn | kolla_url(public_protocol, prometheus_alertmanager_public_port) }}"
+prometheus_internal_endpoint: "{{ prometheus_internal_fqdn | kolla_url(internal_protocol, prometheus_port) }}"
+prometheus_public_endpoint: "{{ prometheus_external_fqdn | kolla_url(public_protocol, prometheus_public_port) }}"
+swift_public_base_endpoint: "{{ swift_external_fqdn | kolla_url(public_protocol, swift_proxy_server_port) }}"

etc/kayobe/kolla/inventory/group_vars/prometheus-blackbox-exporter

@@ -0,0 +1,6 @@
+---
+critical:
+  - |
+    Fixes `CVE-2024-40767
+    <https://security.openstack.org/ossa/OSSA-2024-002.html>`_ with updated
+    container images for Nova services.

releasenotes/notes/fix-cve-2024-40767-24b9b3c35f61a0c8.yaml

@@ -0,0 +1,5 @@
+---
+features:
+  - |
+    ``OVN`` version in Rocky Linux 9 container images has been updated to
+    ``24.03`` (latest LTS).

releasenotes/notes/ovn-24-03-47c1eb9846f261b0.yaml

@@ -0,0 +1,6 @@
+---
+fixes:
+  - |
+    Updates Octavia container images to fix a maintenance task that was
+    breaking OVN IPv4 load balancers with health monitors.
+    `LP#2072754 <https://bugs.launchpad.net/nova/+bug/2072754>`__.

releasenotes/notes/ovn-octavia-provider-bug-2072754-5cbd7dc9c366668d.yaml

@@ -0,0 +1,7 @@
+---
+features:
+  - |
+    Added a new group variable - ``stackhpc_repos_enabled`` - for unified
+    control over usage of StackHPC Release Train package repositories. This
+    makes it easier to set which hosts do or do not pull packages from release
+    train.

releasenotes/notes/stackhpc-repos-enabled-10c8a698991e53c2.yaml

@@ -0,0 +1,4 @@
+---
+fixes:
+  - |
+    Fixes the issue with interface names containing dashes in Hashicorp collection.