Enable CIS benchmark hardening in AIO and Multinode

.github/workflows/stackhpc-all-in-one.yml

@@ -355,6 +355,20 @@ jobs:
           KAYOBE_AUTOMATION_SSH_PRIVATE_KEY: ${{ steps.ssh_key.outputs.ssh_key }}
         if: inputs.upgrade
 
+      - name: Ensure we have IP on breth1 to reach the instances
+        # NOTE(wszumski): Whilst we don't need to create resources again, in some circumstances
+        # we can lose the IP address that allows us to connect to the instances. This playbook
+        # also fixes that issue.
+        run: |
+          docker run -t --rm \
+            -v $(pwd):/stack/kayobe-automation-env/src/kayobe-config \
+            -e KAYOBE_ENVIRONMENT -e KAYOBE_VAULT_PASSWORD -e KAYOBE_AUTOMATION_SSH_PRIVATE_KEY \
+            ${{ steps.kayobe_image.outputs.kayobe_image }} \
+            /stack/kayobe-automation-env/src/kayobe-config/.automation/pipeline/playbook-run.sh etc/kayobe/ansible/configure-aio-resources.yml
+        env:
+          KAYOBE_AUTOMATION_SSH_PRIVATE_KEY: ${{ steps.ssh_key.outputs.ssh_key }}
+        if: inputs.upgrade
+
       - name: Tempest tests
         id: tempest
         run: |

doc/source/configuration/security-hardening.rst

@@ -19,7 +19,7 @@ Configuration
 --------------
 
 Some overrides to the role defaults are provided in
-``$KAYOBE_CONFIG_PATH/inventory/group_vars/overcloud/cis``. These may not be
+``$KAYOBE_CONFIG_PATH/inventory/group_vars/cis-hardening/cis``. These may not be
 suitable for all deployments and so some fine tuning may be required. For
 instance, you may want different rules on a network node compared to a
 controller. It is best to consult the upstream role documentation for details
@@ -31,6 +31,14 @@ about what each variable does. The documentation can be found here:
 Running the playbooks
 ---------------------
 
+.. note::
+
+  The hosts may need rebooting to fully pick up all of the changes. The CIS
+  roles will warn you when this needs to be done, but the actual reboot is left
+  as a manual operation to allow you to select a convenient time. Generally, if
+  you are applying the hardening for the first time, then you will need to
+  reboot.
+
 As there is potential for unintended side effects when applying the hardening
 playbooks, the playbooks are not currently enabled by default. It is recommended
 that they are first applied to a representative staging environment to determine
@@ -40,3 +48,33 @@ whether or not workloads or API requests are affected by any configuration chang
 
     kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/cis.yml
 
+Targetting additional hosts
+---------------------------
+
+The ``cis.yml`` playbook targets hosts in the ``cis-hardening`` group. By
+default this includes the ``overcloud`` group. You can adjust this group
+to suit your needs, e.g to add the seed VM:
+
+.. code-block:: yaml
+  :caption: $KAYOBE_CONFIG_PATH/inventory/groups
+
+  [cis-hardening:children]
+  overcloud
+  seed
+
+Enabling the host configure hook
+--------------------------------
+
+A hook is pre-installed but its execution is guarded by the
+``stackhpc_enable_cis_benchmark_hardening_hook`` configuration option.
+If you want the hardening playbooks to run automatically, as part of
+host configure, simply set this flag to ``true``:
+
+.. code-block:: yaml
+  :caption: $KAYOBE_CONFIG_PATH/stackhpc.yml
+
+    stackhpc_enable_cis_benchmark_hardening_hook: true
+
+Alternatively, this can be toggled on a per-environment basis by
+setting it in an environment specific config file, or even on
+targeted hosts by using group or host vars.

etc/kayobe/ansible/cis.yml

@@ -1,8 +1,10 @@
 ---
 
 - name: Security hardening
-  hosts: overcloud
+  hosts: cis-hardening
   become: true
+  tags:
+    - cis
   tasks:
     # TODO: Remove this when Red Hat FIPS policy has been updated to allow ed25519 keys.
     # https://gitlab.com/gitlab-org/gitlab/-/issues/367429#note_1840422075

etc/kayobe/environments/ci-aio/stackhpc.yml

@@ -0,0 +1,3 @@
+---
+
+stackhpc_enable_cis_benchmark_hardening_hook: true

etc/kayobe/environments/ci-multinode/stackhpc.yml

@@ -0,0 +1,3 @@
+---
+
+stackhpc_enable_cis_benchmark_hardening_hook: true

etc/kayobe/hooks/overcloud-host-configure/post.d/99-cis.yml

@@ -0,0 +1,4 @@
+---
+
+- import_playbook: ../../../ansible/cis.yml
+  when: stackhpc_enable_cis_benchmark_hardening_hook | bool

etc/kayobe/inventory/group_vars/all/stackhpc

@@ -0,0 +1,6 @@
+---
+###############################################################################
+# Feature flags
+
+# Whether or not to run CIS benchmark hardening playbooks. Default is false.
+stackhpc_enable_cis_benchmark_hardening_hook: false

etc/kayobe/inventory/groups

@@ -125,3 +125,9 @@ rgws
 [mgrs]
 [osds]
 [rgws]
+
+###############################################################################
+# Feature control groups
+
+[cis-hardening:children]
+overcloud

etc/kayobe/stackhpc.yml

@@ -152,3 +152,9 @@ stackhpc_docker_registry: "{{ pulp_url | regex_replace('^https?://', '') }}"
 # Username and password of container registry.
 stackhpc_docker_registry_username: "{{ pulp_username }}"
 stackhpc_docker_registry_password: "{{ pulp_password }}"
+
+###############################################################################
+# Feature flags
+
+# Whether or not to run CIS benchmark hardening playbooks. Default is false.
+#stackhpc_enable_cis_benchmark_hardening_hook:

releasenotes/notes/adds-cis-hook-8cec8d42103d075e.yaml

@@ -0,0 +1,7 @@
+---
+features:
+  - |
+    Adds a hook to automatically run the CIS benchmark hardening playbooks as
+    part of host configure. This is guarded by the
+    ``stackhpc_enable_cis_benchmark_hardening_hook`` configuration option and is
+    disabled by default.