Emit metrics on deprecated APIs dynamically #2155

exdx · 2021-05-10T21:15:38Z

Signed-off-by: Daniel Sover [email protected]

Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1958296

Description of the change:

Motivation for the change:

Reviewer Checklist

Implementation matches the proposed design, or proposal is updated to match implementation
Sufficient unit test coverage
Sufficient end-to-end test coverage
Docs updated or added to /doc
Commit messages sensible and descriptive

…s in the installplan Signed-off-by: Daniel Sover <[email protected]>

openshift-ci · 2021-05-10T21:15:40Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

openshift-ci · 2021-05-10T21:15:41Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: exdx
To complete the pull request process, please assign ecordell after the PR has been reviewed.
You can assign the PR to them by writing /assign @ecordell in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci · 2021-05-10T21:15:46Z

@exdx: This pull request references Bugzilla bug 1958296, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.8.0) matches configured target release for branch (4.8.0)
bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @jianzhangbjz

In response to this:

Bug 1958296: emit metrics on deprecated APIs dynamically

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2021-05-10T21:15:47Z

@openshift-ci[bot]: GitHub didn't allow me to request PR reviews from the following users: jianzhangbjz.

Note that only operator-framework members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

@exdx: This pull request references Bugzilla bug 1958296, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

bug is open, matching expected state (open)

bug target release (4.8.0) matches configured target release for branch (4.8.0)

bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @jianzhangbjz

In response to this:

Bug 1958296: emit metrics on deprecated APIs dynamically

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

timflannagan · 2021-05-11T16:51:30Z

I'm going to remove the "Bug <...>" from the title so the BZ automation doesn't pick this up before a downstream PR has been created. I'll update your description to add a link to the BZ instead.

/retitle Emit metrics on deprecated APIs dynamically

openshift-ci · 2021-05-11T16:51:33Z

@exdx: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

In response to this:

Emit metrics on deprecated APIs dynamically

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

timflannagan · 2021-05-11T16:51:36Z

/retest

benluddy · 2021-05-11T18:59:52Z

pkg/lib/scopedclient/warning.go

+	return config
+}
+
+func (w *WarningRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) {


I don't think we can safely read from or close the response body here, at least not without copying to some kind of buffer and setting up a functionally-equivalent ReadCloser on the response before returning it.

Is there a nice way to reliably reverse REST mappings from the request URL path back to a GVK/GVR and name?

Edit: Alternatively, could this copy the warning headers somewhere, then have the step application code (which should know everything about the step it is trying to apply) poll them?

Ah, I was hoping we could do that. But it makes sense, since this is just part of the path of the response, we should not read or close it directly (we could copy the data and do whatever on that though).

I can checkout the idea of reversing the URL to get the resource info (not to unmarshal the response directly).

Putting code into the stepper would work too, but feels like quite a bit of complexity

benluddy · 2021-05-11T19:03:56Z

pkg/metrics/metrics.go

+			Name: "install_plan_warnings_count",
+			Help: "monotonic count of resources created with warnings (for example, deprecation warnings)",
+		},
+		[]string{WARNING_LABEL, NAME_LABEL, NAMESPACE_LABEL, GVK_LABEL},


If the warning text, namespace, name, and GVK are all labels, and the values of those labels come from user activity, the risk of blowing up timeseries cardinality feels high to me.

Can most of the specific information go into an Event (possibly associated with the InstallPlan)?

It's certainly a concern, but since we are just alerting here on deprecated objects on creation I'm wondering where the cardinality would truly explode? Is it cases where the installplan includes additional subscriptions from dependency resolution and you basically have an N dimensional install

I was looking to embed things like the installplan name or even better the operator package in the WarningRoundTripper struct, instantiate it in the ExcutePlan() step with all the info, and then pass it down when creating the clients

benluddy · 2021-05-11T19:37:03Z

pkg/controller/operators/catalog/step.go

@@ -84,6 +88,10 @@ func (b *builder) create(step v1alpha1.Step) (Stepper, error) {
 		case crdlib.V1Version:
 			return b.NewCRDV1Step(b.opclient.ApiextensionsInterface().ApiextensionsV1(), &step, manifest), nil
 		case crdlib.V1Beta1Version:
+			// Emit error about deprecated API
+			// This codepath will be removed entirely in 1.22
+			metrics.EmitInstallPlanWarning(v1beta1DeprecationWarning, step.Resource.Name, "",


Why is this special case necessary?

We install CRDs with OLM's client and the scoped client RoundTripper would not apply

benluddy · 2021-05-11T19:42:25Z

pkg/lib/scopedclient/clientfactory.go

@@ -26,6 +26,7 @@ type Factory struct {
 // token specified.
 func (f *Factory) NewOperatorClient(token string) (client operatorclient.ClientInterface, err error) {
 	scoped := copy(f.config, token)
+	scoped = SetWarningRoundTripper(scoped)


Especially since this is shared between olm and catalog, I think it would be better to inject the warning sniffer instead of setting it up here in scopedclient.

Yep, had the same idea 👍

Signed-off-by: Daniel Sover <[email protected]>

kevinrizza · 2021-05-13T13:09:12Z

pkg/controller/operators/adoption_controller.go

@@ -235,6 +237,14 @@ func (r *AdoptionReconciler) adoptComponents(ctx context.Context, csv *operators
 			}()
 		}

+		// alert on components that are deprecated in upcoming kubernetes versions


Do we need to emit this on every reconciliation? Are we potentially going to start spewing metrics far too frequently in some cases? Can we batch these somehow?

I'm wondering if we should be aggregating this kind of metric and have a separate process that emits them.

Yup, I realized this was an issue with this implementation -- we probably need to move it around somewhere.

Aggregating the metric sounds interest as well, so as not to explode the cardinality?

exdx · 2021-06-01T16:03:47Z

Closing in favor of another PR.

fix: use custom RoundTripper to capture warnings when creating object…

2c099ba

…s in the installplan Signed-off-by: Daniel Sover <[email protected]>

exdx requested review from benluddy and kevinrizza May 10, 2021 21:15

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 10, 2021

openshift-ci bot requested review from dinhxuanvu and njhale May 10, 2021 21:15

openshift-ci bot added bugzilla/severity-urgent Referenced Bugzilla bug's severity is urgent for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. labels May 10, 2021

openshift-ci bot changed the title ~~Bug 1958296: emit metrics on deprecated APIs dynamically~~ Emit metrics on deprecated APIs dynamically May 11, 2021

openshift-ci bot removed bugzilla/severity-urgent Referenced Bugzilla bug's severity is urgent for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. labels May 11, 2021

benluddy reviewed May 11, 2021

View reviewed changes

add deprecation warning to operator adoption controller

34047fd

Signed-off-by: Daniel Sover <[email protected]>

kevinrizza reviewed May 13, 2021

View reviewed changes

exdx closed this Jun 1, 2021

openshift-ci bot mentioned this pull request Jun 9, 2021

Bug 1958296: Surface API server warnings when applying plan steps. openshift/operator-framework-olm#85

Merged

Emit metrics on deprecated APIs dynamically #2155

Emit metrics on deprecated APIs dynamically #2155

Uh oh!

Conversation

exdx commented May 10, 2021 • edited by timflannagan Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci bot commented May 10, 2021

Uh oh!

openshift-ci bot commented May 10, 2021

Uh oh!

openshift-ci bot commented May 10, 2021

Uh oh!

openshift-ci bot commented May 10, 2021

Uh oh!

timflannagan commented May 11, 2021

Uh oh!

openshift-ci bot commented May 11, 2021

Uh oh!

timflannagan commented May 11, 2021

Uh oh!

benluddy May 11, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

exdx commented Jun 1, 2021

Uh oh!

Uh oh!

exdx commented May 10, 2021 •

edited by timflannagan

Loading

benluddy May 11, 2021 •

edited

Loading