Skip to content

Commit de30039

Browse files
zzhloginzzhlogin
authored
Add main_build and release_build workflow (#39)
*Issue #, if available:* This PR add three workflows: 1. pr_build: with all the PR action in main & release branches, build project .whl file and image file for future contract tests. This workflow is based on [Java_pr_build](https://github.com/aws-observability/aws-otel-java-instrumentation/blob/main/.github/workflows/pr-build.yml#L80) with following difference: a. No `testpatch`: Have cut a backlog item to have the infrastructure set up for it. b. os: Although [otel java setup three matrix.os](https://github.com/aws-observability/aws-otel-java-instrumentation/blob/fb25e6d5589abe7ad2985c6c277a10b82bbc15af/.github/workflows/pr-build.yml#L46), it is actually running the following actions [only on ubuntu-latest](https://github.com/aws-observability/aws-otel-java-instrumentation/blob/fb25e6d5589abe7ad2985c6c277a10b82bbc15af/.github/workflows/pr-build.yml#L80). We need different command for different type of OS, which cause many discrepancies, and the workflow won’t pass. So, we will just do linux for now. Also cut a backlog item for mac/windows support. c. No `test-adot-javaagent-image.sh`: we currently don't have[ .github/scripts/test-adot-javaagent-image.sh](https://github.com/aws-observability/aws-otel-java-instrumentation/blob/main/.github/scripts/test-adot-javaagent-image.sh)(it use to compare [the docker built image](https://github.com/aws-observability/aws-otel-java-instrumentation/blob/cf3c27aa183cf361fee39176a2e5e278bf363185/.github/actions/cpUtility-testing/action.yml#L46) with [the Gradle built image](https://github.com/aws-observability/aws-otel-java-instrumentation/blob/cf3c27aa183cf361fee39176a2e5e278bf363185/.github/workflows/main-build.yml#L92)). Since python don't have build tool like Gradle, we cannot do comparison like Java d. No contract tests, will be added when it is ready. e. Added codespell check. f. run on python version [py38, py39, py310, py311] same as the upstream [distro](https://github.com/open-telemetry/opentelemetry-python-contrib/blob/8fd2105ceae604cf39a67f1c4dd154753b43fcd1/.github/workflows/instrumentations_0.yml#L25C9-L25C58) 3. main_build: with all the push action and main & release branch, build project .whl file upload to staging S3 bucket, and build project image, upload to staging ECR repo. This workflow is based on [Java_main_build](https://github.com/aws-observability/aws-otel-java-instrumentation/blob/main/.github/workflows/main-build.yml) with following difference: a. No `testpatch`, see above in pr_build for details. b. No `test-adot-javaagent-image.sh`: see above in pr_build for details. c. No E2E tests, contract tests, publish-build-status, will be added when they are ready. 4. release_build: when the workflow triggered manually, build project .whl file and public it, also build project image, upload to staging ECR repo, and publish to Test PyPI and PyPI. Currently we are using private ECR repo only with release tag. When the public ECR is ready, we will add it there. This workflow is based on [Java_release_build](https://github.com/aws-observability/aws-otel-java-instrumentation/blob/main/.github/workflows/release-build.yml) with following difference: a. No `test-adot-javaagent-image.sh`: see above in pr_build for details. b. Added publish to Test PyPI and PyPI. Comment out publish PyPI for now, will be uncomment when release workflow tested. Sample successfully workflow runs: PR build: https://github.com/aws-observability/aws-otel-python-instrumentation/actions/runs/7880276267/job/21502034322?pr=39 main build: https://github.com/aws-observability/aws-otel-python-instrumentation/actions/runs/7880275859/job/21502033292?pr=39 By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
1 parent 41a3c60 commit de30039

File tree

8 files changed

+387
-79
lines changed

8 files changed

+387
-79
lines changed

‎.github/actions/artifacts_build/action.yml

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
name: Build and Push aws-opentelemetry-distro Wheel and Image files according inputs
2+
description: |
3+
This action assumes that Repo was checked out and Python was set correctly
4+
5+
inputs:
6+
aws-region:
7+
required: true
8+
description: "AWS Region"
9+
image_uri_with_tag:
10+
required: true
11+
description: "Image URI with Tag"
12+
image_registry:
13+
required: true
14+
description: "Image Registry"
15+
snapshot-ecr-role:
16+
require: true
17+
description: "IAM Role used for pushing to snapshot ecr"
18+
push_image:
19+
required: true
20+
description: "Whether push image to ECR"
21+
load_image:
22+
required: true
23+
description: "Whether load the image for the following use, only load in PR build"
24+
python_version:
25+
required: true
26+
description: "The python version used in actions"
27+
package_name:
28+
required: true
29+
description: "The package name"
30+
os:
31+
required: true
32+
description: "The os"
33+
34+
35+
runs:
36+
using: "composite"
37+
steps:
38+
- name: Set up Python
39+
uses: actions/setup-python@v4
40+
with:
41+
python-version: ${{ matrix.python-version }}
42+
43+
- name: Install tox
44+
shell: bash
45+
run: pip install tox==3.27.1 tox-factor
46+
47+
- name: Cache tox environment
48+
# Preserves .tox directory between runs for faster installs
49+
uses: actions/cache@v1
50+
with:
51+
path: |
52+
.tox
53+
~/.cache/pip
54+
key: v7-build-tox-cache-${{ matrix.python-version }}-${{ inputs.package_name }}-${{ inputs.os }}-${{ hashFiles('tox.ini', 'dev-requirements.txt') }}
55+
56+
- name: run tox
57+
shell: bash
58+
run: tox -f ${{ matrix.python-version }}-${{ inputs.package_name }} -- -ra --benchmark-json=${{ matrix.python-version }}-${{ inputs.package_name }}-${{ inputs.os }}-benchmark.json
59+
60+
- name: Configure AWS Credentials
61+
uses: aws-actions/configure-aws-credentials@v4
62+
with:
63+
role-to-assume: ${{ inputs.snapshot-ecr-role }}
64+
aws-region: ${{ inputs.aws-region }}
65+
66+
- name: Install Dependencies and Build Wheel
67+
id: staging_wheel_build
68+
shell: bash
69+
run: |
70+
pip install --upgrade pip setuptools wheel packaging build
71+
rm -rf ./dist/*
72+
cd ./aws-opentelemetry-distro
73+
python -m build --outdir ../dist
74+
75+
- name: Set up QEMU
76+
uses: docker/setup-qemu-action@v3
77+
78+
- name: Set up Docker Buildx
79+
uses: docker/setup-buildx-action@v3
80+
81+
- name: Login to AWS ECR
82+
uses: docker/login-action@v3
83+
with:
84+
registry: ${{ inputs.image_registry }}
85+
env:
86+
AWS_REGION: ${{ inputs.aws-region }}
87+
88+
- name: Build and push image according to input
89+
uses: docker/build-push-action@v5
90+
with:
91+
push: ${{ inputs.push_image }}
92+
context: .
93+
file: ./Dockerfile
94+
platforms: linux/amd64
95+
tags: ${{ inputs.image_uri_with_tag }}
96+
load: ${{ inputs.load_image }}

‎.github/workflows/main_build.yml

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
# This workflow build the aws-opentelemetry-distro wheel file, upload to staging S3 bucket, and build project docker image then push to staging ECR
2+
name: Python Instrumentation Main Build
3+
on:
4+
push:
5+
branches:
6+
- main
7+
- "release/v*"
8+
env:
9+
AWS_DEFAULT_REGION: us-east-1
10+
STAGING_ECR_REGISTRY: 637423224110.dkr.ecr.us-east-1.amazonaws.com
11+
STAGING_ECR_REPOSITORY: aws-observability/adot-autoinstrumentation-python-staging
12+
S3_INTEGRATION_BUCKET: ${{ secrets.S3_INTEGRATION_BUCKET }}
13+
14+
concurrency:
15+
group: python-instrumentation-main-build
16+
cancel-in-progress: false
17+
18+
permissions:
19+
id-token: write
20+
contents: read
21+
22+
jobs:
23+
build:
24+
runs-on: ubuntu-latest
25+
outputs:
26+
python_image_tag: ${{ steps.python_output.outputs.python_image_tag}}
27+
staging_image: ${{ steps.python_output.outputs.stagingImage}}
28+
staging_registry: ${{ steps.python_output.outputs.stagingRegistry}}
29+
staging_repository: ${{ steps.python_output.outputs.stagingRepository}}
30+
staging_wheel_file: ${{ steps.staging_wheel_output.outputs.STAGING_WHEEL}}
31+
steps:
32+
- name: Checkout Contrib Repo @ SHA - ${{ github.sha }}
33+
uses: actions/checkout@v4
34+
35+
- name: Get Python Distro Output
36+
id: python_output
37+
run: |
38+
pkg_version=$(grep '__version__' ./aws-opentelemetry-distro/src/amazon/opentelemetry/distro/version.py | awk -F '"' '{print $2}')
39+
echo "ADOT_PYTHON_VERSION=$pkg_version" >> $GITHUB_OUTPUT
40+
shortsha="$(git rev-parse --short HEAD)"
41+
echo "SHORT_SHA=$pkg_version" >> $GITHUB_ENV
42+
python_distro_tag=$pkg_version-$shortsha
43+
echo "python_image_tag=$python_distro_tag" >> $GITHUB_OUTPUT
44+
echo "stagingRegistry=${{ env.STAGING_ECR_REGISTRY }}" >> $GITHUB_OUTPUT
45+
echo "stagingRepository=${{ env.STAGING_ECR_REPOSITORY }}" >> $GITHUB_OUTPUT
46+
echo "stagingImage=${{ env.STAGING_ECR_REGISTRY }}/${{ env.STAGING_ECR_REPOSITORY }}:$python_distro_tag" >> $GITHUB_OUTPUT
47+
48+
- name: Build and Push Wheel and Image Files
49+
uses: ./.github/actions/artifacts_build
50+
with:
51+
aws-region: ${{ env.AWS_DEFAULT_REGION }}
52+
image_uri_with_tag: ${{ steps.python_output.outputs.stagingImage}}
53+
image_registry: ${{ env.STAGING_ECR_REGISTRY }}
54+
snapshot-ecr-role: ${{ secrets.AWS_ASSUME_ROLE_ARN }}
55+
push_image: true
56+
load_image: false
57+
python_version: 3.11
58+
package_name: aws-opentelemetry-distro
59+
os: ubuntu-latest
60+
61+
- name: Output Wheel File Name
62+
id: staging_wheel_output
63+
run: |
64+
staging_wheel="aws_opentelemetry_distro-${{ steps.python_output.outputs.ADOT_PYTHON_VERSION}}-${{ env.SHORT_SHA }}-py3-none-any.whl"
65+
echo "STAGING_WHEEL=$staging_wheel" >> $GITHUB_OUTPUT
66+
cd ./dist
67+
cp aws_opentelemetry_distro-${{ steps.python_output.outputs.ADOT_PYTHON_VERSION}}-py3-none-any.whl $staging_wheel
68+
69+
- name: Upload wheel to S3
70+
run: |
71+
aws s3 cp dist/${{ steps.staging_wheel_output.outputs.STAGING_WHEEL}} ${{ env.S3_INTEGRATION_BUCKET }}
72+
73+
- name: Upload Wheel to GitHub Actions
74+
uses: actions/upload-artifact@v3
75+
with:
76+
name: aws_opentelemetry_distro-${{ steps.python_output.outputs.ADOT_PYTHON_VERSION}}-py3-none-any.whl
77+
path: dist/${{ steps.staging_wheel_output.outputs.STAGING_WHEEL}}
78+
79+
# TODO: Add Contract test and E2E test

‎.github/workflows/pr_build.yml

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
name: Python Instrumentation PR Build
2+
on:
3+
pull_request:
4+
branches:
5+
- main
6+
- "release/v*"
7+
8+
env:
9+
AWS_DEFAULT_REGION: us-east-1
10+
TEST_TAG: 637423224110.dkr.ecr.us-east-1.amazonaws.com/eks/observability/adot-autoinstrumentation-python:test
11+
12+
permissions:
13+
id-token: write
14+
contents: read
15+
16+
jobs:
17+
build:
18+
runs-on: ubuntu-latest
19+
strategy:
20+
matrix:
21+
python-version: ["3.8", "3.9", "3.10", "3.11"]
22+
tox-environment: ["spellcheck", "lint"]
23+
steps:
24+
- name: Checkout Contrib Repo @ SHA - ${{ github.sha }}
25+
uses: actions/checkout@v4
26+
27+
- name: Install libsnappy-dev
28+
if: ${{ matrix.tox-environment == 'lint' }}
29+
run: sudo apt-get update && sudo apt-get install -y libsnappy-dev
30+
31+
- name: Build Wheel and Image Files
32+
uses: ./.github/actions/artifacts_build
33+
with:
34+
aws-region: ${{ env.AWS_DEFAULT_REGION }}
35+
image_uri_with_tag: ${{ env.TEST_TAG }}
36+
image_registry: 637423224110.dkr.ecr.us-east-1.amazonaws.com
37+
snapshot-ecr-role: ${{ secrets.AWS_ASSUME_ROLE_ARN }}
38+
push_image: false
39+
load_image: true
40+
python_version: ${{ matrix.python-version }}
41+
package_name: aws-opentelemetry-distro
42+
os: ubuntu-latest
43+
44+
- name: run spell check tox
45+
run: tox -e ${{ matrix.tox-environment }}
46+
47+
# TODO: Add Contract test.

‎.github/workflows/release_build.yml

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
#This Release Build workflow is not fully ready yet, will be tested as Beta release when it is ready.
2+
name: Release Build
3+
on:
4+
workflow_dispatch:
5+
inputs:
6+
version:
7+
description: The version to tag the release with, e.g., 1.2.0, 1.2.1-alpha.1
8+
required: true
9+
10+
env:
11+
AWS_DEFAULT_REGION: us-east-1
12+
ECR_REGISTRY: 637423224110.dkr.ecr.us-east-1.amazonaws.com
13+
PRIVATE_ECR_REPOSITORY: eks/observability/adot-autoinstrumentation-python
14+
15+
permissions:
16+
id-token: write
17+
contents: write
18+
19+
jobs:
20+
build:
21+
runs-on: ubuntu-latest
22+
steps:
23+
- name: Checkout Contrib Repo @ SHA - ${{ github.sha }}
24+
uses: actions/checkout@v4
25+
26+
- name: Build Wheel and Image Files
27+
uses: ./.github/actions/artifacts_build
28+
with:
29+
aws-region: ${{ env.AWS_DEFAULT_REGION }}
30+
image_uri_with_tag: ${{ env.ECR_REGISTRY }}/${{ env.PRIVATE_ECR_REPOSITORY }}:v${{ github.event.inputs.version }}
31+
image_registry: ${{ env.ECR_REGISTRY }}
32+
snapshot-ecr-role: ${{ secrets.AWS_ASSUME_ROLE_ARN }}
33+
push_image: false
34+
load_image: false
35+
python_version: 3.11
36+
package_name: aws-opentelemetry-distro
37+
os: ubuntu-latest
38+
39+
- name: Get PyPI secrets by name and ARN
40+
uses: aws-actions/aws-secretsmanager-get-secrets@v1
41+
id: pypi_secrets
42+
with:
43+
secret-ids: |
44+
PROD_PYPI_TOKEN,arn:aws:secretsmanager:us-east-1:637423224110:secret:prod/PyPI/apiToken-W2a9ny
45+
TEST_PYPI_TOKEN,arn:aws:secretsmanager:us-east-1:637423224110:secret:test/PyPI/apiToken-z5iqc6
46+
47+
- name: Export distro version
48+
id: distro_version
49+
shell: bash
50+
run: |
51+
pkg_version=$(grep '__version__' ./aws-opentelemetry-distro/src/amazon/opentelemetry/distro/version.py | awk -F '"' '{print $2}')
52+
echo "ADOT_PYTHON_VERSION=$pkg_version" >> $GITHUB_OUTPUT
53+
54+
# The step below publishes to testpypi in order to catch any issues
55+
# with the package configuration that would cause a failure to upload to pypi.
56+
- name: Publish to TestPyPI
57+
env:
58+
TWINE_USERNAME: '__token__'
59+
TWINE_PASSWORD: ${{ steps.pypi_secrets.outputs.TEST_PYPI_TOKEN }}
60+
run: |
61+
pip install --upgrade twine
62+
twine upload --repository testpypi --skip-existing --verbose dist/aws_opentelemetry_distro-${{ steps.distro_version.outputs.ADOT_PYTHON_VERSION }}-py3-none-any.whl
63+
64+
- name: Push release image
65+
uses: docker/build-push-action@v5
66+
with:
67+
push: true
68+
context: .
69+
file: ./Dockerfile
70+
platforms: linux/amd64,linux/arm64
71+
tags: ${{ env.ECR_REGISTRY }}/${{ env.PRIVATE_ECR_REPOSITORY }}:v${{ github.event.inputs.version }}
72+
73+
#TODO: Uncomment "Publish to PyPI" after test release_build workflow.
74+
# - name: Publish to PyPI
75+
# env:
76+
# TWINE_USERNAME: '__token__'
77+
# TWINE_PASSWORD: ${{ steps.pypi_secrets.outputs.PROD_PYPI_TOKEN }}
78+
# run: |
79+
# twine upload --skip-existing --verbose dist/aws_opentelemetry_distro-${{ steps.distro_version.outputs.ADOT_PYTHON_VERSION }}-py3-none-any.whl
80+
81+
- name: Create release
82+
id: create_release
83+
env:
84+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} # This token is provided by Actions, you do not need to create your own token
85+
run: |
86+
gh release create --target "$GITHUB_REF_NAME" \
87+
--title "Release v${{ github.event.inputs.version }}" \
88+
--draft \
89+
"v${{ github.event.inputs.version }}" \
90+
dist/aws_opentelemetry_distro-${{ steps.distro_version.outputs.ADOT_PYTHON_VERSION }}-py3-none-any.whl

‎.github/workflows/test.yml

Lines changed: 0 additions & 73 deletions
This file was deleted.

0 commit comments

Comments
 (0)