Skip to content

Sync 07/26 #343

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 16 commits into from
Jul 28, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions api.blacklist
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
decf74800a179da9f002ae2234204c7e6b866f9a
45 changes: 23 additions & 22 deletions docs/downstream-ci.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,62 +4,63 @@ The CI configuration for each release branch can be found [here](https://github.
From `4.11` (`master` as of this writing) we've updated the configuration to able to influence CI on a PR basis. An overview of the `ci-operator` (the system used for ci)
can be found [here](https://docs.ci.openshift.org/docs/architecture/ci-operator/).

### Structure
## Structure

* `.ci-operator.yaml` defines the `build_root_image`. To be ART compliant, the image should come from the [ocp-build-data](https://github.com/openshift/ocp-build-data/) repo
* [openshift-operator-framework-olm-master.yaml](https://github.com/openshift/release/blob/master/ci-operator/config/openshift/operator-framework-olm/openshift-operator-framework-olm-master.yaml) defines the images that are used by ci, produced by ci, and the ci jobs the get executed.
* `base.Dockerfile` defines the image used by ci to execute the ci jobs
- `.ci-operator.yaml` defines the `build_root_image`. To be ART compliant, the image should come from the [ocp-build-data](https://github.com/openshift/ocp-build-data/) repo
- [openshift-operator-framework-olm-master.yaml](https://github.com/openshift/release/blob/master/ci-operator/config/openshift/operator-framework-olm/openshift-operator-framework-olm-master.yaml) defines the images that are used by ci, produced by ci, and the ci jobs the get executed.
- `base.Dockerfile` defines the image used by ci to execute the ci jobs

From [openshift-operator-framework-olm-master.yaml](https://github.com/openshift/release/blob/master/ci-operator/config/openshift/operator-framework-olm/openshift-operator-framework-olm-master.yaml), we see under the `images` stanza the `ci-image` definition.
It goes from `src` (the `build_root_image`) to `ci-image` by building `base.Dockerfile` with `src` as the base image.

```
```yaml
- dockerfile_path: base.Dockerfile
from: src
to: ci-image
```

The image is excluded from promotion, to never be posted up anywhere:

```
```yaml
promotion:
excluded_images:
- ci-image
```

and each `test` references `ci-image` as the image to be used to the test, e.g.:

```
```yaml
tests:
- as: verify
commands: make verify
container:
from: ci-image
```

### Updating go versions
## Updating go versions

All we need to do is update the `build_root_image` referenced in `.ci-operator.yaml` and we may also need to update the `base_images` in [openshift-operator-framework-olm-master.yaml](https://github.com/openshift/release/blob/master/ci-operator/config/openshift/operator-framework-olm/openshift-operator-framework-olm-master.yaml).
All we need to do is update the `build_root_image` referenced in `.ci-operator.yaml` and we may also need to update the `base_images` in [openshift-operator-framework-olm-master.yaml](https://github.com/openshift/release/blob/master/ci-operator/config/openshift/operator-framework-olm/openshift-operator-framework-olm-master.yaml).

**NOTE**: I believe there is some automation that updates the base images, though I don't know. I'll leave this as a questions to the reviewer, and if no one knows, I'll go after it.

### Downstream sync
## Downstream sync

The complete information about the downstreaming process can be found [here](https://docs.google.com/document/d/139yXeOqAJbV1ndC7Q4NbaOtzbSdNpcuJan0iemORd3g/edit).

TL;DR;

We sync three upstream repositories ([api](https://github.com/operator-framework/api), [registry](https://github.com/operator-framework/operator-registry), [olm](https://github.com/operator-framework/operator-lifecycle-manager)) to the downstream [olm mono-repo](https://github.com/openshift/operator-framework-olm). Commits from the upstream repositories are cherry-picked to the appropriate `staging` directory in the downstream repository. Because this is a monorepo in the `Openshift` GitHub organization, two things need to be remembered:
- we don't pull in upstream `vendor` folder changes
- we don't pull in changes to `OWNERS` files
- after each cherry-pick we execute: `make vendor` and `make manifests` to ensure a) the downstream dependencies are updated b) to ensure any manifest changes are picked up downstream
-- Note: `make manifests` requires [GNU sed](https://www.gnu.org/software/sed/)

While manual changes to the `staging` directory should be avoided, there could be instances where there drift between the downstream `staging` directory and the corresponding upstream repository. This can happen due to applying commits out-of-order, e.g. due to feature freeze, etc.
- we don't pull in upstream `vendor` folder changes
- we don't pull in changes to `OWNERS` files
- after each cherry-pick we execute: `make vendor` and `make manifests` to ensure a) the downstream dependencies are updated b) to ensure any manifest changes are picked up downstream
- Note: `make manifests` requires [GNU sed](https://www.gnu.org/software/sed/)

While manual changes to the `staging` directory should be avoided, there could be instances where there drift between the downstream `staging` directory and the corresponding upstream repository. This can happen due to applying commits out-of-order, e.g. due to feature freeze, etc.

Therefore, after a sync, it is important to manually verify the diff of `staging` and the upstream. Please note, though, that some downstream changes are downstream only. These are, however, few and far between and there are comments to indicate that a block of code is downstream only.
Therefore, after a sync, it is important to manually verify the diff of `staging` and the upstream. Please note, though, that some downstream changes are downstream only. These are, however, few and far between and there are comments to indicate that a block of code is downstream only.

The downstream sync process is facilitated by two scripts: `scripts/sync_get_candidates.sh` and `scripts/sync_pop_candidate.sh`, which compare the upstream remote with the appropriate `staging` directory and gets a stack of commits to sync, and cherry-pick those commits in reverse order. What does this look like in practice:
The downstream sync process is facilitated by two scripts: `scripts/sync_get_candidates.sh` and `scripts/sync_pop_candidate.sh`, which compare the upstream remote with the appropriate `staging` directory and gets a stack of commits to sync, and cherry-pick those commits in reverse order. What does this look like in practice:

```bash
# Clone downstream
Expand All @@ -72,8 +73,8 @@ git remote add operator-lifecycle-manager [email protected]:operator-framework/oper

# Get upstream commit candidates: ./scripts/sync_get_candidates.sh <api|operator-registry|operator-lifecycle-manager> <branch>
# The shas will be found in ./<api|operator-registry|operator-lifecycle-manager>.cherrypick
./scripts/sync_get_candidates.sh api master
./scripts/sync_get_candidates.sh operator-registry master
./scripts/sync_get_candidates.sh api master
./scripts/sync_get_candidates.sh operator-registry master
./scripts/sync_get_candidates.sh operator-lifecycle-manager master

# Sync upstream commits: ./scripts/sync_pop_candidate.sh <api|operator-registry|operator-lifecycle-manager> [-a]
Expand All @@ -90,7 +91,7 @@ sync_pop_candidate.sh operator-registry -a
# Depending on the changes being pulled in, the order of repos you sync _could_ matter and _could_ leave a commit in an unbuildable state
```

Example:
Example:

```bash
$ sync_pop_candidate.sh operator-lifecycle-manager -a
Expand All @@ -117,7 +118,7 @@ hint: run "git cherry-pick --abort".

$ rm -rf staging/operator-lifecycle-manager/vendor

# make sure there are no conflics in
# make sure there are no conflics in
# staging/operator-lifecycle-manager/go.mod and go.sum
$ cd staging/operator-lifecycle-manager
$ go mod tidy
Expand All @@ -131,4 +132,4 @@ $ sync_pop_candidate.sh operator-lifecycle-manager -a

#### Running console test locally

The [console](https://github.com/openshift/console) repository contains all instructions you need to execute the console tests locally. The olm console tests can be found [here](https://github.com/openshift/console/tree/master/frontend/packages/operator-lifecycle-manager)
The [console](https://github.com/openshift/console) repository contains all instructions you need to execute the console tests locally. The olm console tests can be found [here](https://github.com/openshift/console/tree/master/frontend/packages/operator-lifecycle-manager)
12 changes: 6 additions & 6 deletions docs/local-testing-with-crc.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Local testing with CRC

We can use CRC as an Openshift-like environment within which to test downstream OLM. [CRC](https://developers.redhat.com/products/codeready-containers/overview)
We can use CRC as an Openshift-like environment within which to test downstream OLM. [CRC](https://developers.redhat.com/products/codeready-containers/overview)
is a tool for deploying a local Openshift cluster on your laptop.

TL;DR
Expand All @@ -10,16 +10,16 @@ TL;DR
3. `export KUBECONFIG=~/.crc/machines/crc/kubeconfig`
4. Execute e2e tests as you normally would, e.g., `make e2e/olm`

#### Gosh darn it, how does it work?
## Gosh darn it, how does it work?

`./scripts/crc-start.sh` is used to provision a crc cluster. `./scripts/crc-deploy.sh` pushes the `olm:test` and `opm:test` to
`image-registry.openshift-image-registry.svc:5000/openshift/olm:test` and `image-registry.openshift-image-registry.svc:5000/openshift/opm:test`
images to the crc image registry under the global project `openshift` (to be independent from the olm namespace). It also generates an image stream
from these images. Finally, using the istag for the image. It also generates the olm manifests by applying generated helm values file (`values-crc-e2e.yaml`)
from these images. Finally, using the istag for the image. It also generates the olm manifests by applying generated helm values file (`values-crc-e2e.yaml`)
and other generated yaml patches (`scripts/*.crc.e2e.patch.yaml`) to make sure the manifests point to the newly pushed images. The generated manifests are
then applied to the cluster using `kubectl replace` in priority order (lexical sort).

#### Make targets
## Make targets

1. `make crc-start`: provision a crc cluster, if necessary
1. `FORCE_CLEAN=1 make crc-start`: nuke any current installation including cache and current cluster instance
Expand All @@ -29,7 +29,7 @@ then applied to the cluster using `kubectl replace` in priority order (lexical s
2. `SKIP_WAIT_READY=1 make crc-deploy`: skip waiting for olm deployments to be available at the end
4. `make crc`: the same as `make crc-start crc-build crc-deploy`

#### Manipulating Resources
## Manipulating Resources

If new resources are introduced that require being updated for local deployment (e.g. updating the pod spec image) follow
the pattern used in `scripts/crc-deploy.sh:make_manifest_patches`
the pattern used in `scripts/crc-deploy.sh:make_manifest_patches`.
5 changes: 2 additions & 3 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ require (
github.com/grpc-ecosystem/grpc-health-probe v0.4.11
github.com/maxbrunsfeld/counterfeiter/v6 v6.4.1
github.com/mikefarah/yq/v3 v3.0.0-20201202084205-8846255d1c37
github.com/onsi/ginkgo v1.16.5
github.com/onsi/ginkgo/v2 v2.1.3
github.com/openshift/api v0.0.0-20200331152225-585af27e34fd
github.com/operator-framework/api v0.15.0
github.com/operator-framework/operator-lifecycle-manager v0.0.0-00010101000000-000000000000
Expand Down Expand Up @@ -124,6 +124,7 @@ require (
github.com/google/gnostic v0.5.7-v3refs // indirect
github.com/google/go-cmp v0.5.6 // indirect
github.com/google/gofuzz v1.2.0 // indirect
github.com/google/pprof v0.0.0-20210720184732-4bb14d4b1be1 // indirect
github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 // indirect
github.com/google/uuid v1.2.0 // indirect
github.com/gorilla/mux v1.8.0 // indirect
Expand Down Expand Up @@ -172,7 +173,6 @@ require (
github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00 // indirect
github.com/morikuni/aec v1.0.0 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/nxadm/tail v1.4.8 // indirect
github.com/onsi/gomega v1.18.1 // indirect
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/opencontainers/image-spec v1.0.3-0.20211202183452-c5a74bcca799 // indirect
Expand Down Expand Up @@ -239,7 +239,6 @@ require (
gopkg.in/natefinch/lumberjack.v2 v2.0.0 // indirect
gopkg.in/op/go-logging.v1 v1.0.0-20160211212156-b2cb9fa56473 // indirect
gopkg.in/square/go-jose.v2 v2.6.0 // indirect
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 // indirect
gopkg.in/warnings.v0 v0.1.2 // indirect
gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b // indirect
k8s.io/apiextensions-apiserver v0.24.0 // indirect
Expand Down
5 changes: 4 additions & 1 deletion go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -635,8 +635,10 @@ github.com/google/pprof v0.0.0-20201023163331-3e6fc7fc9c4c/go.mod h1:kpwsk12EmLe
github.com/google/pprof v0.0.0-20201203190320-1bf35d6f28c2/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE=
github.com/google/pprof v0.0.0-20210122040257-d980be63207e/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE=
github.com/google/pprof v0.0.0-20210226084205-cbba55b83ad5/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE=
github.com/google/pprof v0.0.0-20210407192527-94a9f03dee38/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE=
github.com/google/pprof v0.0.0-20210601050228-01bbb1931b22/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE=
github.com/google/pprof v0.0.0-20210609004039-a478d1d731e9/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE=
github.com/google/pprof v0.0.0-20210720184732-4bb14d4b1be1 h1:K6RDEckDVWvDI9JAJYCmNdQXq6neHJOYx3V6jnqNEec=
github.com/google/pprof v0.0.0-20210720184732-4bb14d4b1be1/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE=
github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI=
github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 h1:El6M4kTTCOh6aBiKaUGG7oYTSPP8MxqL4YI3kZKwcP4=
Expand Down Expand Up @@ -951,7 +953,8 @@ github.com/onsi/ginkgo v1.12.1/go.mod h1:zj2OWP4+oCPe1qIXoGWkgMRwljMUYCdkwsT2108
github.com/onsi/ginkgo v1.14.0/go.mod h1:iSB4RoI2tjJc9BBv4NKIKWKya62Rps+oPG/Lv9klQyY=
github.com/onsi/ginkgo v1.16.4/go.mod h1:dX+/inL/fNMqNlz0e9LfyB9TswhZpCVdJM/Z6Vvnwo0=
github.com/onsi/ginkgo v1.16.5 h1:8xi0RTUf59SOSfEtZMvwTvXYMzG4gV23XVHOZiXNtnE=
github.com/onsi/ginkgo v1.16.5/go.mod h1:+E8gABHa3K6zRBolWtd+ROzc/U5bkGt0FwiG042wbpU=
github.com/onsi/ginkgo/v2 v2.1.3 h1:e/3Cwtogj0HA+25nMP1jCMDIf8RtRYbGwGGuBIFztkc=
github.com/onsi/ginkgo/v2 v2.1.3/go.mod h1:vw5CSIxN1JObi/U8gcbwft7ZxR2dgaR70JSE3/PpL4c=
github.com/onsi/gomega v1.17.0 h1:9Luw4uT5HTjHTN8+aNcSThgH1vdXnmdJ8xIfZ4wyTRE=
github.com/onsi/gomega v1.17.0/go.mod h1:HnhC7FXeEQY45zxNK3PPoIUhzk/80Xly9PcubAlGdZY=
github.com/opencontainers/go-digest v0.0.0-20170106003457-a6d0ee40d420/go.mod h1:cMLVZDEM3+U2I4VmLI6N8jQYUd2OVphdqWwCJHrFt2s=
Expand Down
1 change: 0 additions & 1 deletion operator-lifecycle-manager.blacklist
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
604825027920cdc52a489669b22f51910cabfa69
fd3322a2bafedce1821ed45b8a9bcce429126af7
ef91a92858f22a2510e243efc2ae114272b6ad7b
c7b6534da9ec82b0266fdd69da8ade80d1a8ff80
1 change: 1 addition & 0 deletions operator-registry.blacklist
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
0b2f43f426f9d621cab7a5224fc260ae3265be1c
eb2e6393796f397e93618c4aa5038f6a366cdb31
4 changes: 2 additions & 2 deletions staging/api/.github/workflows/go.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ jobs:
- name: Set up Go
uses: actions/setup-go@v2
with:
go-version: 1.17
go-version: 1.18
id: go
- name: Check out code into the Go module directory
uses: actions/checkout@v2
Expand All @@ -39,7 +39,7 @@ jobs:
- name: Set up Go
uses: actions/setup-go@v2
with:
go-version: 1.17
go-version: 1.18
id: go
- name: Print out Go env
run: go env
Expand Down
2 changes: 1 addition & 1 deletion staging/api/.github/workflows/verify.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ jobs:
- uses: actions/checkout@v2
- uses: actions/setup-go@v2
with:
go-version: '~1.17'
go-version: '~1.18'
- name: Run the verify target
run: |
export GOPATH=$(go env GOPATH)
Expand Down
51 changes: 26 additions & 25 deletions staging/api/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -86,29 +86,30 @@ test-unit: ## Run the unit tests
verify: manifests generate format
git diff --exit-code

###
# Utilities.
###
################
# Hack / Tools #
################

GO_INSTALL_OPTS ?= "-mod=mod"

## Location to install dependencies to
LOCALBIN ?= $(shell pwd)/bin
$(LOCALBIN):
mkdir -p $(LOCALBIN)

## Tool Binaries
CONTROLLER_GEN ?= $(LOCALBIN)/controller-gen
YQ ?= $(LOCALBIN)/yq

## Tool Versions
CONTROLLER_TOOLS_VERSION ?= v0.8.0

.PHONY: controller-gen
controller-gen: $(CONTROLLER_GEN) ## Download controller-gen locally if necessary.
$(CONTROLLER_GEN): $(LOCALBIN)
GOBIN=$(LOCALBIN) go install $(GO_INSTALL_OPTS) sigs.k8s.io/controller-tools/cmd/controller-gen@$(CONTROLLER_TOOLS_VERSION)

# go-get-tool will 'go get' any package $2 and install it to $1.
PROJECT_DIR := $(shell dirname $(abspath $(lastword $(MAKEFILE_LIST))))
define go-get-tool
@[ -f $(1) ] || { \
set -e ;\
TMP_DIR=$$(mktemp -d) ;\
cd $$TMP_DIR ;\
go mod init tmp ;\
echo "Downloading $(2)" ;\
GOBIN=$(PROJECT_DIR)/bin go get $(2) ;\
GOBIN=$(PROJECT_DIR)/bin go install $(2) ;\
rm -rf $$TMP_DIR ;\
}
endef

CONTROLLER_GEN = $(shell pwd)/bin/controller-gen
controller-gen: ## Download controller-gen locally if necessary.
$(call go-get-tool,$(CONTROLLER_GEN),sigs.k8s.io/controller-tools/cmd/[email protected])

YQ = $(shell pwd)/bin/yq
yq:
$(call go-get-tool,$(YQ),github.com/mikefarah/yq/v3)
.PHONY: yq
yq: $(YQ) ## Download yq locally if necessary.
$(YQ): $(LOCALBIN)
GOBIN=$(LOCALBIN) go install $(GO_INSTALL_OPTS) github.com/mikefarah/yq/v3@latest
2 changes: 1 addition & 1 deletion staging/api/go.mod
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
module github.com/operator-framework/api

go 1.17
go 1.18

require (
github.com/blang/semver v3.5.1+incompatible
Expand Down
Loading