Skip to content

Nightly Slurm CI Rocky update workflow #440

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 50 commits into from
Oct 10, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
03732bc
Update openstack.pkr.hcl
bertiethorpe Sep 16, 2024
06ee0f2
new image build workflow
bertiethorpe Sep 16, 2024
d6d72ba
dynamically set packer vars from fatimage workflow
bertiethorpe Sep 17, 2024
ce2afdb
remove openstack. prefix from image name
bertiethorpe Sep 17, 2024
edbaeb7
echo image name
bertiethorpe Sep 17, 2024
47391ad
make image_name var in packer config
bertiethorpe Sep 17, 2024
92e6d1c
new changes
bertiethorpe Sep 18, 2024
e36dad7
Merge branch 'main' into ci/nightly-builds
bertiethorpe Sep 18, 2024
ac08548
fix merge changes
bertiethorpe Sep 18, 2024
101f696
temp workflow changes
bertiethorpe Sep 18, 2024
cf8a685
test nightly build
bertiethorpe Sep 18, 2024
26cd493
change back fatimage workflow
bertiethorpe Sep 18, 2024
cf37a32
rename images built
bertiethorpe Sep 19, 2024
c21d097
add update to builder group
bertiethorpe Sep 19, 2024
64918cb
add update to fatimage build groups
bertiethorpe Sep 19, 2024
8e71d6a
fatimage.yml fix
bertiethorpe Sep 19, 2024
990ddc1
move output image_name declaration into build blocks
bertiethorpe Sep 20, 2024
aa0786f
delete outdated nightly image
bertiethorpe Sep 20, 2024
6defc9d
test new fatimage build
bertiethorpe Sep 20, 2024
c29a9db
debug dnf remove cockpit
bertiethorpe Sep 24, 2024
3b601f5
--amend
bertiethorpe Sep 24, 2024
f3d0b67
add cuda build back in
bertiethorpe Sep 24, 2024
35048ac
cuda nightly build
bertiethorpe Sep 26, 2024
d16ef50
test cuda nightly builds
bertiethorpe Sep 26, 2024
b98d803
test new fatimage build on SMS
bertiethorpe Sep 27, 2024
79f6893
test image upploads across clouds
bertiethorpe Sep 27, 2024
7183fcc
test image uploads in separate workflow
bertiethorpe Sep 30, 2024
02fa0b6
finish nightly build workflow
bertiethorpe Sep 30, 2024
260146a
fix image delete logic
bertiethorpe Sep 30, 2024
fd44029
use azimuth-cloud trivy db mirror
bertiethorpe Oct 1, 2024
716352b
use GITHUB_TOKEN env
bertiethorpe Oct 1, 2024
cfa40d3
test new fatimage build
bertiethorpe Oct 1, 2024
5c13b41
Merge branch 'main' into ci/nightly-builds
bertiethorpe Oct 2, 2024
6bffdf7
add final nightlybuilds workflow
bertiethorpe Oct 2, 2024
348c150
move trivy scan to separate workflow
bertiethorpe Oct 3, 2024
6c37ac8
bump image and test new trivy scan
bertiethorpe Oct 3, 2024
a256bce
fix artifact creation
bertiethorpe Oct 3, 2024
3fbc9e9
bump image and test trivy scan
bertiethorpe Oct 3, 2024
a60dbf4
only run trivy scan on image bumps
bertiethorpe Oct 3, 2024
d883e55
bump image to test trivy scan run condition
bertiethorpe Oct 4, 2024
00eaf4e
bump cuda image
bertiethorpe Oct 4, 2024
8630ebb
bump image
bertiethorpe Oct 4, 2024
46c1c15
extend timeout for trivy scanning cuda image
bertiethorpe Oct 4, 2024
22dcfbb
Run workflow on PR to main
bertiethorpe Oct 4, 2024
b61ee98
Merge branch 'main' into ci/nightly-builds
bertiethorpe Oct 4, 2024
c9ebf42
address PR comments
bertiethorpe Oct 8, 2024
527f7bc
fix source_image_name packer parse
bertiethorpe Oct 9, 2024
cf417e5
bump image
bertiethorpe Oct 9, 2024
354b048
additional PR comments
bertiethorpe Oct 9, 2024
169b42d
bump image
bertiethorpe Oct 9, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
107 changes: 44 additions & 63 deletions .github/workflows/fatimage.yml
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@

name: Build fat image
on:
workflow_dispatch:
inputs:
ci_cloud:
description: 'Select the CI_CLOUD'
required: true
type: choice
options:
- LEAFCLOUD
- SMS
- ARCUS
inputs:
ci_cloud:
description: 'Select the CI_CLOUD'
required: true
type: choice
options:
- LEAFCLOUD
- SMS
- ARCUS

jobs:
openstack:
name: openstack-imagebuild
Expand All @@ -25,7 +25,7 @@ jobs:
- RL8
- RL9
build:
- openstack.openhpc-ofed
- openstack.openhpc
- openstack.openhpc-cuda
exclude:
- os_version: RL8
Expand All @@ -34,6 +34,18 @@ jobs:
ANSIBLE_FORCE_COLOR: True
OS_CLOUD: openstack
CI_CLOUD: ${{ github.event.inputs.ci_cloud }}
SOURCE_IMAGES_MAP: |
{
"RL8": {
"openstack.openhpc": "rocky-latest-RL8",
"openstack.openhpc-cuda": "rocky-latest-cuda-RL8"
},
"RL9": {
"openstack.openhpc": "rocky-latest-RL9",
"openstack.openhpc-cuda": "rocky-latest-cuda-RL9"
}
}

steps:
- uses: actions/checkout@v2

Expand All @@ -52,10 +64,10 @@ jobs:
- name: Add bastion's ssh key to known_hosts
run: cat environments/.stackhpc/bastion_fingerprints >> ~/.ssh/known_hosts
shell: bash

- name: Install ansible etc
run: dev/setup-env.sh

- name: Write clouds.yaml
run: |
mkdir -p ~/.config/openstack/
Expand All @@ -66,17 +78,25 @@ jobs:
run: |
. venv/bin/activate
. environments/.stackhpc/activate

- name: Build fat image with packer
id: packer_build
run: |
set -x
. venv/bin/activate
. environments/.stackhpc/activate
cd packer/
packer init .
PACKER_LOG=1 packer build -on-error=${{ vars.PACKER_ON_ERROR }} -only=${{ matrix.build }} -var-file=$PKR_VAR_environment_root/${{ env.CI_CLOUD }}.pkrvars.hcl openstack.pkr.hcl

PACKER_LOG=1 packer build \
-on-error=${{ vars.PACKER_ON_ERROR }} \
-only=${{ matrix.build }} \
-var-file=$PKR_VAR_environment_root/${{ env.CI_CLOUD }}.pkrvars.hcl \
-var "source_image_name=${{ env.SOURCE_IMAGE }}" \
openstack.pkr.hcl
env:
PKR_VAR_os_version: ${{ matrix.os_version }}
SOURCE_IMAGE: ${{ fromJSON(env.SOURCE_IMAGES_MAP)[matrix.os_version][matrix.build] }}

- name: Get created image names from manifest
id: manifest
Expand All @@ -87,53 +107,14 @@ jobs:
sleep 5
done
IMAGE_NAME=$(openstack image show -f value -c name $IMAGE_ID)
echo "image-name=${IMAGE_NAME}" >> "$GITHUB_OUTPUT"
echo "image-id=$IMAGE_ID" >> "$GITHUB_OUTPUT"

- name: Download image
run: |
. venv/bin/activate
sudo mkdir /mnt/images
sudo chmod 777 /mnt/images
openstack image save --file /mnt/images/${{ steps.manifest.outputs.image-name }}.qcow2 ${{ steps.manifest.outputs.image-name }}

- name: Set up QEMU
uses: docker/setup-qemu-action@v3

- name: install libguestfs
run: |
sudo apt -y update
sudo apt -y install libguestfs-tools

- name: mkdir for mount
run: sudo mkdir -p './${{ steps.manifest.outputs.image-name }}'

- name: mount qcow2 file
run: sudo guestmount -a /mnt/images/${{ steps.manifest.outputs.image-name }}.qcow2 -i --ro -o allow_other './${{ steps.manifest.outputs.image-name }}'

- name: Run Trivy vulnerability scanner
uses: aquasecurity/[email protected]
with:
scan-type: fs
scan-ref: "${{ steps.manifest.outputs.image-name }}"
scanners: "vuln"
format: sarif
output: "${{ steps.manifest.outputs.image-name }}.sarif"
# turn off secret scanning to speed things up

- name: Upload Trivy scan results to GitHub Security tab
uses: github/codeql-action/upload-sarif@v3
with:
sarif_file: "${{ steps.manifest.outputs.image-name }}.sarif"
category: "${{ matrix.os_version }}-${{ matrix.build }}"
echo $IMAGE_ID > image-id.txt
echo $IMAGE_NAME > image-name.txt

- name: Fail if scan has CRITICAL vulnerabilities
uses: aquasecurity/[email protected]
- name: Upload manifest artifact
uses: actions/upload-artifact@v4
with:
scan-type: fs
scan-ref: "${{ steps.manifest.outputs.image-name }}"
scanners: "vuln"
format: table
exit-code: '1'
severity: 'CRITICAL'
ignore-unfixed: true
name: image-details-${{ matrix.build }}-${{ matrix.os_version }}
path: |
./image-id.txt
./image-name.txt
overwrite: true
Loading
Loading