Add flake analyzer periodics #1645

Bowenislandsong · 2020-07-16T17:33:05Z

This commit introduces the flake analyzer to OLM to create weekly flake reports for the past and previous 7 days. The result will be uploaded to GITHUB as artifacts.

Description of the change:

Motivation for the change:

Reviewer Checklist

Implementation matches the proposed design, or proposal is updated to match implementation
Sufficient unit test coverage
Sufficient end-to-end test coverage
Docs updated or added to /docs
Commit messages sensible and descriptive

Bowenislandsong · 2020-07-16T17:33:30Z

/hold
Wait for a success run

Bowenislandsong · 2020-07-16T17:35:59Z

/cc @ecordell
Waiting for personal access token named FLAKE_BOT_SECRET

Bowenislandsong · 2020-07-16T21:21:12Z

No personal access token needed. weekly report test run was successful and availble at https://github.com/operator-framework/operator-lifecycle-manager/runs/879483265?check_suite_focus=true

Bowenislandsong · 2020-07-16T21:48:35Z

/hold cancel
Periodic report generated https://github.com/operator-framework/operator-lifecycle-manager/actions/runs/171922622
This periodic report will be available per day which daily, and weekly reports on each run.

The current implementation can not accommodate PR commenting due to token issue. When the analyzer becomes a bot, we will be able to comment.

Bowenislandsong · 2020-07-17T01:34:54Z

/retest

Bowenislandsong · 2020-07-17T11:24:14Z

/retest

Bowenislandsong · 2020-07-17T13:49:43Z

/retest

Bowenislandsong · 2020-07-17T15:14:25Z

/retest

Bowenislandsong · 2020-07-17T15:51:40Z

/hold
Adding comment abaility

This commit introduces the flake analyzer to OLM to create weekly flake reports for the past and previous 7 days plus today. The result will be uploaded to GITHUB as artifacts.

Bowenislandsong · 2020-07-17T22:56:17Z

/hold cancel
comment ability is avaliable via any existing workflow add ons. Bot with a server or periodics are the only options.

Bowenislandsong · 2020-07-18T00:22:05Z

The PR failed tests for 0 times with 0 individual failed tests and 0 skipped tests.

totaltestcount: 0
flaketestcount: 0
skippedtestcount: 0
flaketests: []
skippedtests: []

Bowenislandsong · 2020-07-18T11:01:02Z

The PR failed tests for 1 times with 1 individual failed tests and 5 skipped tests.

totaltestcount: 1
flaketestcount: 1
skippedtestcount: 5
flaketests:

classname: End-to-end
name: Subscription creation manual approval
counts: 1
details:

count: 1
error: |4-

/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/subscription_e2e_test.go:178

	Error Trace:	subscription_e2e_test.go:185
	            				runner.go:113
	            				runner.go:64
	            				it_node.go:26
	            				spec.go:215
	            				spec.go:138
	            				spec_runner.go:200
	            				spec_runner.go:170
	            				spec_runner.go:66
	            				suite.go:62
	            				ginkgo_dsl.go:226
	            				ginkgo_dsl.go:214
	            				e2e_test.go:54
	Error:      	Received unexpected error:
	            	E2E bug detected: configmaps "mock-ocs" already exists

/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/github.com/stretchr/testify/require/require.go:1005

meandurationsec: 0.040457
skippedtests:

classname: End-to-end
name: Subscriptions create required objects from Catalogs Given a Namespace when
a CatalogSource is created with a bundle that contains prometheus objects
creating a subscription using the CatalogSource should have created the expected
prometheus objects
counts: 1
details: []
meandurationsec: 2.16539
classname: End-to-end
name: Subscriptions create required objects from Catalogs Given a Namespace when
a CatalogSource is created with a bundle that contains prometheus objects
creating a subscription using the CatalogSource should install the operator
successfully
counts: 1
details: []
meandurationsec: 2.926843
classname: End-to-end
name: Subscription updates existing install plan
counts: 1
details: []
meandurationsec: 0.095173
classname: End-to-end
name: Installing bundles with new object types when a bundle with a pdb, priorityclass,
and VPA object is installed should create the additional bundle objects
counts: 1
details: []
meandurationsec: 0
classname: End-to-end
name: Catalog image update
counts: 1
details: []
meandurationsec: 0.014702

Bowenislandsong · 2020-07-18T14:09:56Z

/retest

Bowenislandsong · 2020-07-19T02:47:11Z

This PR failed tests for 1 times with 1 individual failed tests and 5 skipped tests. A test is considered flaky if failed on multiple commits.

totaltestcount: 1
flaketestcount: 1
skippedtestcount: 5
flaketests:

classname: End-to-end
name: 'Install Plan with CRD schema change Test all existing versions are present,
different (backwards compatible) schema'
counts: 1
details:

count: 1
error: |4-

/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/github.com/onsi/ginkgo/extensions/table/table_entry.go:43

	Error Trace:	installplan_e2e_test.go:812
	            				value.go:460
	            				value.go:321
	            				table_entry.go:37
	            				runner.go:113
	            				runner.go:64
	            				it_node.go:26
	            				spec.go:215
	            				spec.go:138
	            				spec_runner.go:200
	            				spec_runner.go:170
	            				spec_runner.go:66
	            				suite.go:62
	            				ginkgo_dsl.go:226
	            				ginkgo_dsl.go:214
	            				e2e_test.go:54
	Error:      	Received unexpected error:
	            	etcdserver: request timed out

/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/github.com/stretchr/testify/require/require.go:1005

meandurationsec: 25.831612
skippedtests:

classname: End-to-end
name: 'Subscription updates existing install plan'
counts: 1
details: []
meandurationsec: 0.023219
classname: End-to-end
name: 'Subscriptions create required objects from Catalogs Given a Namespace
when a CatalogSource is created with a bundle that contains prometheus objects
creating a subscription using the CatalogSource should install the operator
successfully'
counts: 1
details: []
meandurationsec: 2.196919
classname: End-to-end
name: 'Subscriptions create required objects from Catalogs Given a Namespace
when a CatalogSource is created with a bundle that contains prometheus objects
creating a subscription using the CatalogSource should have created the expected
prometheus objects'
counts: 1
details: []
meandurationsec: 2.215525
classname: End-to-end
name: 'Installing bundles with new object types when a bundle with a pdb, priorityclass,
and VPA object is installed should create the additional bundle objects'
counts: 1
details: []
meandurationsec: 0
classname: End-to-end
name: 'Catalog image update'
counts: 1
details: []
meandurationsec: 0.023103

Bowenislandsong · 2020-07-20T13:11:52Z

/retest

.github/workflows/e2e-tests.yml

benluddy · 2020-07-22T14:54:25Z

/approve

openshift-ci-robot · 2020-07-22T14:54:29Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: benluddy, Bowenislandsong

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [benluddy]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ecordell · 2020-07-22T14:58:11Z

/lgtm

openshift-bot · 2020-07-22T15:38:49Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-07-22T17:35:48Z

/retest

Please review the full test history for this PR and help us cut down flakes.

Bowenislandsong · 2020-07-22T19:38:35Z

/retest

openshift-ci-robot requested review from awgreene and ecordell July 16, 2020 17:33

openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 16, 2020

Bowenislandsong force-pushed the feat/add-flake-analyzer branch 2 times, most recently from 1a86433 to b506c5c Compare July 16, 2020 21:01

openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 16, 2020

Bowenislandsong force-pushed the feat/add-flake-analyzer branch 2 times, most recently from a500dbe to 8dda9ad Compare July 16, 2020 21:52

openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 17, 2020

Bowenislandsong force-pushed the feat/add-flake-analyzer branch from 89f4096 to 1da2805 Compare July 17, 2020 16:25

Add flake analyzer

1e8e896

This commit introduces the flake analyzer to OLM to create weekly flake reports for the past and previous 7 days plus today. The result will be uploaded to GITHUB as artifacts.

Bowenislandsong force-pushed the feat/add-flake-analyzer branch 7 times, most recently from 0576004 to 1e8e896 Compare July 17, 2020 22:53

openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 17, 2020

Bowenislandsong changed the title ~~Add flake analyzer~~ Add flake analyzer periodics Jul 20, 2020

benluddy reviewed Jul 22, 2020

View reviewed changes

.github/workflows/e2e-tests.yml Show resolved Hide resolved

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 22, 2020

openshift-ci-robot assigned ecordell Jul 22, 2020

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Jul 22, 2020

openshift-merge-robot merged commit cc2112c into operator-framework:master Jul 22, 2020

Add flake analyzer periodics #1645

Add flake analyzer periodics #1645

Uh oh!

Conversation

Bowenislandsong commented Jul 16, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Bowenislandsong commented Jul 16, 2020

Uh oh!

Bowenislandsong commented Jul 16, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Bowenislandsong commented Jul 16, 2020

Uh oh!

Bowenislandsong commented Jul 16, 2020

Uh oh!

Bowenislandsong commented Jul 17, 2020

Uh oh!

Bowenislandsong commented Jul 17, 2020

Uh oh!

Bowenislandsong commented Jul 17, 2020

Uh oh!

Bowenislandsong commented Jul 17, 2020

Uh oh!

Bowenislandsong commented Jul 17, 2020

Uh oh!

Bowenislandsong commented Jul 17, 2020

Uh oh!

Bowenislandsong commented Jul 18, 2020

Uh oh!

Bowenislandsong commented Jul 18, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Bowenislandsong commented Jul 18, 2020

Uh oh!

Bowenislandsong commented Jul 19, 2020

Uh oh!

Bowenislandsong commented Jul 20, 2020

Uh oh!

Uh oh!

benluddy commented Jul 22, 2020

Uh oh!

openshift-ci-robot commented Jul 22, 2020

Uh oh!

ecordell commented Jul 22, 2020

Uh oh!

openshift-bot commented Jul 22, 2020

Uh oh!

openshift-bot commented Jul 22, 2020

Uh oh!

Bowenislandsong commented Jul 22, 2020

Uh oh!

Uh oh!

Bowenislandsong commented Jul 16, 2020 •

edited

Loading

Bowenislandsong commented Jul 16, 2020 •

edited

Loading

Bowenislandsong commented Jul 18, 2020 •

edited

Loading