[test] Fix workspace integration tests #17222

kylos101 · 2023-04-14T21:50:06Z

Description

Stabilize workspace integration tests (we're down to one failure now, TestGitActions, which intermittently fails), which we'll tackle in a separate PR.

This required that we:

update the packer image to the version being used in gen95 (WKS-107). The image contained many fixes (including a bump to containerd 1.6.20, K3s 1.26, etc.). These updates help avoid disk pressure issues, which was causing tests to fail intermittently. 💡 I suspect this was also leading to preview environments being unstable in general.
make code changes in /test to account for recent product changes (like organizationId being required for createWorkspace with server)
leverage the trust-manager manifests built into the packer image, and modify the startup script, so that it retries applying trust-manager manifests to workaround timing issues causing preview environment failures.

Also, I've added /test sub-folders to .github/CODEOWNERS. This should help teams (1) inspect changes (2) assert tests pass before merge to main and (3) collaborate on testing.

How to test

git checkout -b some-name to a local branch, and then build a preview TF_VAR_infra_provider="gce" TF_VAR_with_large_vm=true leeway run dev:preview (Why? The branch for this PR is rate limited until May 2 at 6pm EST via Let's Encrypt.)
run the workspace integration tests from kylos101/gen94-integ branch, with asset kylos101-gen94-integ-gha.8496. Here's a sample run that passed ✔️ .

Release Notes

NONE

Documentation

Build Options:

/werft with-werft
Run the build with werft instead of GHA
leeway-no-cache
/werft no-test
Run Leeway with --dont-test

Publish Options

/werft publish-to-npm
/werft publish-to-jb-marketplace

Installer Options

with-dedicated-emulation
with-ws-manager-mk2
workspace-feature-flags
Add desired feature flags to the end of the line above, space separated

Preview Environment Options:

/werft with-local-preview
If enabled this will build install/preview
/werft with-preview
/werft with-large-vm
/werft with-gce-vm
If enabled this will create the environment on GCE infra
with-integration-tests=all
Valid options are all, workspace, webapp, ide, jetbrains, vscode, ssh

/hold

kylos101 · 2023-04-24T21:52:23Z

@mustard-mh and @akosyakov for 👀 , because I'm changing integration tests (workspace tests have been horribly broken), and adding /test sub-folders to CODEOWNERS. Will hopefully be marking this PR as ready for review soon. 🙏

akosyakov · 2023-04-25T09:22:00Z

On our side if running ide as target works then it is alright. When PR is not draft we are happy to review. @iQQBot looked quite a lot in integration testing, for sure can help 🙏

.github/CODEOWNERS

test/README.md

kylos101 · 2023-04-28T18:52:48Z

The trust-manager fix in action:

root@kylos101-gc38a6d66fe:/home/kyle# journalctl -t install-trustmanager -xf
Apr 28 18:51:26 kylos101-gc38a6d66fe install-trustmanager[3784]: Starting to install trust manager
Apr 28 18:51:26 kylos101-gc38a6d66fe install-trustmanager[3818]: Sleeping...
Apr 28 18:51:32 kylos101-gc38a6d66fe install-trustmanager[4988]: Trust manager applied
Apr 28 18:51:32 kylos101-gc38a6d66fe install-trustmanager[5039]: Sleeping...
Apr 28 18:51:38 kylos101-gc38a6d66fe install-trustmanager[5929]: Trust manager applied
Apr 28 18:51:38 kylos101-gc38a6d66fe install-trustmanager[5955]: Sleeping...
Apr 28 18:51:43 kylos101-gc38a6d66fe install-trustmanager[6497]: Trust manager applied
Apr 28 18:51:43 kylos101-gc38a6d66fe install-trustmanager[6523]: Sleeping...
Apr 28 18:51:48 kylos101-gc38a6d66fe install-trustmanager[7101]: Trust manager applied
Apr 28 18:52:11 kylos101-gc38a6d66fe install-trustmanager[8364]: Finishing installing trust manager

kylos101 · 2023-04-28T19:57:00Z

Can build preview envs:

TF_VAR_infra_provider="gce" TF_VAR_with_large_vm=true leeway run dev:preview
SUCCESS: Installation is happy: https://kylos101-gc38a6d66fe.preview.gitpod-dev.com/workspaces

Might remove later...

It's expected on the Typescript side: https://github.com/gitpod-io/gitpod/blob/90b7d658589fd6e9fb239ff239591b9f1218fc83/components/server/src/workspace/gitpod-server-impl.ts#L1227-L1235

But sometimes there's no team 🤷

We use UBP now, there is no more unleashed. Also, remove the "ff" feature flag code (which was for PVC). It was mutating the username, resulting in Code 460 errors on createWorkspace

Tests intermittently fail with to avoid intermittent failures

This way, we can assert tests are passing for all teams prior to merging

Test to see if flakeyness goes away... ...and bump the timeout because we reduced parallel runs

This: 1. updates from K3s 1.23 to 1.26 2. requires that we remove PodSecurityPolicy changes (as it's no longer supported) 3. resolves intermittent disk pressure issues

* We were getting PSP from rook/ceph, which I think was for PVC * We were getting PSP from the monitoring-satellite

…git actions. Why? We miss state transitions, it's not guaranteed each one will be returned, and there are other tests waiting. For example, in the below log, we miss INITIALIZING, RUNNING, and STOPPING. workspace.go:369: attempt to create the workspace as user 0565bb3c-e724-4da9-84fb-22e2a7b23b8c, with context github.com/gitpod-io/gitpod-test-repo/tree/integration-test/commit workspace.go:411: attempt to get the workspace information: gitpodio-gitpodtestrepo-nscsowy1njb workspace.go:423: not preparing workspace.go:432: got the workspace information: gitpodio-gitpodtestrepo-nscsowy1njb workspace.go:460: wait for workspace to be fully up and running workspace.go:569: prepare for a connection with ws-manager workspace.go:590: established for a connection with ws-manager workspace.go:598: check if the status of workspace is in the running phase: 462f1325-3019-4547-8666-508e8353335e workspace.go:631: status: 462f1325-3019-4547-8666-508e8353335e, PENDING workspace.go:598: check if the status of workspace is in the running phase: 462f1325-3019-4547-8666-508e8353335e workspace.go:631: status: 462f1325-3019-4547-8666-508e8353335e, PENDING workspace.go:598: check if the status of workspace is in the running phase: 462f1325-3019-4547-8666-508e8353335e workspace.go:631: status: 462f1325-3019-4547-8666-508e8353335e, CREATING workspace.go:598: check if the status of workspace is in the running phase: 462f1325-3019-4547-8666-508e8353335e workspace.go:631: status: 462f1325-3019-4547-8666-508e8353335e, CREATING workspace.go:598: check if the status of workspace is in the running phase: 462f1325-3019-4547-8666-508e8353335e workspace.go:631: status: 462f1325-3019-4547-8666-508e8353335e, CREATING workspace.go:598: check if the status of workspace is in the running phase: 462f1325-3019-4547-8666-508e8353335e workspace.go:504: waiting for stopping the workspace: 462f1325-3019-4547-8666-508e8353335e workspace.go:514: attemp to delete the workspace: 462f1325-3019-4547-8666-508e8353335e workspace.go:797: confirmed the worksapce is stopped: 462f1325-3019-4547-8666-508e8353335e, STOPPED workspace.go:538: successfully terminated workspace git_test.go:172: failed to wait for the workspace to start up: cannot wait for workspace: context deadline exceeded

And use trust-manager from the packer image

kylos101 · 2023-05-03T20:06:13Z

One test intermittently failed here: TestOpenWorkspaceFromOutdatedPrebuild/prebuild/it_should_open_a_workspace_from_with_an_older_prebuild_initializer_successfully_and_run_the_init_task/classic: github.com/gitpod-io/gitpod/test/tests/components/ws-manager

Will follow-up in a separate PR.

roboquat added release-note-none do-not-merge/hold do-not-merge/work-in-progress size/L labels Apr 14, 2023

kylos101 force-pushed the kylos101/gen94-integ branch 5 times, most recently from dc44fa9 to 39ef66b Compare April 21, 2023 19:07

kylos101 force-pushed the kylos101/gen94-integ branch 2 times, most recently from 08a7e5c to 23ce740 Compare April 24, 2023 21:25

iQQBot reviewed Apr 25, 2023

View reviewed changes

.github/CODEOWNERS Outdated Show resolved Hide resolved

test/README.md Outdated Show resolved Hide resolved

kylos101 force-pushed the kylos101/gen94-integ branch from 23ce740 to 61fa273 Compare April 26, 2023 02:19

roboquat added size/XL and removed size/L labels Apr 26, 2023

kylos101 force-pushed the kylos101/gen94-integ branch from f293a1c to 8a85a26 Compare April 26, 2023 17:34

roboquat added size/XXL and removed size/XL labels Apr 26, 2023

kylos101 force-pushed the kylos101/gen94-integ branch 5 times, most recently from 0aee955 to 6b27225 Compare April 28, 2023 18:45

kylos101 force-pushed the kylos101/gen94-integ branch 2 times, most recently from fb33798 to b9a68fb Compare April 28, 2023 19:37

kylos101 force-pushed the kylos101/gen94-integ branch from b9a68fb to 87bd305 Compare April 28, 2023 20:23

kylos101 added 21 commits May 3, 2023 18:32

[test] bypass exit code 1 on git commit with --allow-empty

99aa47c

[test] Fix gcloud auth, do setup before auth

a3b4937

Show all output

ca75e37

Might remove later...

[test] avoid using deleted user, identity, and token

7d60c28

[test] add organizationId to CreateWorkspaceOptions

c216ff6

It's expected on the Typescript side: https://github.com/gitpod-io/gitpod/blob/90b7d658589fd6e9fb239ff239591b9f1218fc83/components/server/src/workspace/gitpod-server-impl.ts#L1227-L1235

[test] orgId is required on createWorkspace

4f66add

But sometimes there's no team 🤷

[test] fix git context tests

0bd754e

We use UBP now, there is no more unleashed. Also, remove the "ff" feature flag code (which was for PVC). It was mutating the username, resulting in Code 460 errors on createWorkspace

[test] Use example test as the example

1257b4c

[test] fix context tests when run as gitpod-integration-test user

5f59a1a

[test] clean-up

05a9bb3

[test] wait for workspaces to stop

242093e

Tests intermittently fail with to avoid intermittent failures

[test] add code owners

9b01eb4

This way, we can assert tests are passing for all teams prior to merging

[test] limit # of tests that can run in parallel

fe3c251

[test] no parallel tests

4be1123

Test to see if flakeyness goes away... ...and bump the timeout because we reduced parallel runs

[preview] update the VM image to have parity with production

15aa2c8

This: 1. updates from K3s 1.23 to 1.26 2. requires that we remove PodSecurityPolicy changes (as it's no longer supported) 3. resolves intermittent disk pressure issues

[preview] no PSP in support of VM image update

96b3e84

* We were getting PSP from rook/ceph, which I think was for PVC * We were getting PSP from the monitoring-satellite

[preview] retry installing trust-manager

d1c504a

And use trust-manager from the packer image

[test] clarify USER_TOKEN value for preview environments

16b64f5

Cleanup

4655cee

[preview] remove commented out yaml related to PodSecurityPolicy

a10d7d5

kylos101 force-pushed the kylos101/gen94-integ branch from 81d9407 to a10d7d5 Compare May 3, 2023 18:33

kylos101 removed the do-not-merge/hold label May 3, 2023

roboquat merged commit 99eb259 into main May 3, 2023

roboquat deleted the kylos101/gen94-integ branch May 3, 2023 20:07

roboquat added deployed: webapp Meta team change is running in production deployed: IDE IDE change is running in production deployed: workspace Workspace team change is running in production deployed Change is completely running in production labels May 15, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[test] Fix workspace integration tests #17222

[test] Fix workspace integration tests #17222

Uh oh!

kylos101 commented Apr 14, 2023 •

edited

Loading

Uh oh!

kylos101 commented Apr 24, 2023

Uh oh!

akosyakov commented Apr 25, 2023

Uh oh!

Uh oh!

Uh oh!

kylos101 commented Apr 28, 2023

Uh oh!

kylos101 commented Apr 28, 2023

Uh oh!

kylos101 commented May 3, 2023

Uh oh!

Uh oh!

[test] Fix workspace integration tests #17222

[test] Fix workspace integration tests #17222

Uh oh!

Conversation

kylos101 commented Apr 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related

How to test

Release Notes

Documentation

Build Options:

Preview Environment Options:

Uh oh!

kylos101 commented Apr 24, 2023

Uh oh!

akosyakov commented Apr 25, 2023

Uh oh!

Uh oh!

Uh oh!

kylos101 commented Apr 28, 2023

Uh oh!

kylos101 commented Apr 28, 2023

Uh oh!

kylos101 commented May 3, 2023

Uh oh!

Uh oh!

kylos101 commented Apr 14, 2023 •

edited

Loading