Skip to content

Grant a workflow permission to upload files to S3

Jump to bottom
Huy Do edited this page Nov 19, 2024 · 3 revisions

This short wiki explains the step to grant a workflow the permission to upload files to S3. A common use case is to upload JSON records to an S3 bucket so that they can be ingested into a ClickHouse table for querying.

The first step is to consult with PyTorch Dev Infra if you need a new S3 bucket or if it's ok to reuse an existing one. If a new bucket is needed, you will need to submit a PR to create one at https://github.com/pytorch-labs/pytorch-gha-infra/blob/main/runners/s3_bucket.tf. After the change is committed, run https://github.com/pytorch-labs/pytorch-gha-infra/actions/workflows/runners-on-dispatch-release.yml workflow to deploy the bucket.

There are two ways to grant the workflows permission to upload files to S3. The permission could be at the runner level where all jobs handling by the runner can do the upload or it could be granted at the workflow level where only specific workflows with the permission can do so.

Grant write permission to S3 at the runner level

You will need to submit 2 PRs for runners on Meta and LF fleets:

  1. Meta runners, i.e. https://github.com/pytorch-labs/pytorch-gha-infra/pull/533,
  2. LF runners, i.e. https://github.com/pytorch/ci-infra/pull/296 aw An alternative approach is to create a new OIDC role (or edit an existing one) to grant the same permission, i.e. https://github.com/pytorch-labs/pytorch-gha-infra/pull/358. Then, the new role can be used in your workflow, i.e. https://github.com/pytorch/executorch/pull/2449. Grant CH the permission to read from the bucket, i.e. https://github.com/pytorch-labs/pytorch-gha-infra/pull/536. This is not needed if the bucket is public.
Clone this wiki locally