OCPBUGS-38290: [release-4.16] (fix) Resolver: list CatSrc using client, instead of referring to registry-server cache #838
Conversation
…istry-server cache (#3349)
Using "available CatalogSources" information from the registry-client cache was causing
cache inconsistency problems.
This has showed up multiple times in production environments over the years, manifesting
itself in the form of the all subscriptions in a namespace being transitioned into an error
state when a Catalogsource that the cache claims to exist, has actually been deleted from the
cluster, but the cache was not updated.
The Subscriptions are transitioned to an error state because of the deleted catalogsource
with the follwing error message:
"message": "failed to populate resolver cache from source <deleted-catalogsource>: failed to list
bundles: rpc error: code = Unavailable desc = connection error: desc = \"transport:
Error while dialing dial tcp: lookup <deleted-catalogsource>.<ns>.svc on 172.....: no such host\"",
"reason": "ErrorPreventedResolution",
"status": "True",
"type": "ResolutionFailed"
This PR switches the information lookup from the cache, to using a client to list the CatalogSources
present in the cluster.
Upstream-repository: operator-lifecycle-manager
Upstream-commit: ff9084a24b19848c02c2cd4b7d827e057f7f8b11
openshift-ci
bot
added
the
approved
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: anik120, tmshort The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/jira refresh |
|
@perdasilva: No Jira issue is referenced in the title of this pull request. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
@anik120 you may need to edit the PR title removing the space preceding the ':' |
anik120
changed the title
openshift-ci-robot
added
jira/severity-important
|
@anik120: This pull request references Jira Issue OCPBUGS-38290, which is invalid:
Comment The bug has been updated to refer to the pull request using the external bug tracker. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/jira refresh |
|
@anik120: This pull request references Jira Issue OCPBUGS-38290, which is invalid:
Comment In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/jira refresh |
|
@anik120: This pull request references Jira Issue OCPBUGS-38290, which is invalid:
Comment In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
@perdasilva thanks for catching that! Ticket looks like is linked correctly now, just have to wait for the 4.17 bug to be verified. |
|
/jira refresh |
|
@anik120: This pull request references Jira Issue OCPBUGS-38290, which is invalid:
Comment In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/label cherry-pick-approved |
openshift-ci
bot
added
cherry-pick-approved
|
/jira refresh |
|
@perdasilva: This pull request references Jira Issue OCPBUGS-38290, which is invalid:
Comment In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/label backport-risk-assessed |
openshift-ci
bot
added
the
backport-risk-assessed
openshift-ci
bot
assigned
bandrade,
emmajiafan,
jianzhangbjz,
KeenonLee,
kuiwang02 and
Xia-Zhao-rh
|
/jira refresh |
openshift-ci-robot
added
jira/valid-bug
|
@anik120: This pull request references Jira Issue OCPBUGS-38290, which is valid. The bug has been moved to the POST state. 7 validation(s) were run on this bug
Requesting review from QA contact: In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/cherry-pick release-4.15 |
|
@anik120: once the present PR merges, I will cherry-pick it on top of release-4.15 in a new PR and assign it to you. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/retest unit-olm |
|
@anik120: The
The following commands are available to trigger optional jobs:
Use
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/test unit-olm |
|
@anik120: all tests passed! Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
openshift-merge-bot
bot
merged commit f9c0881
into
openshift:release-4.16
|
@anik120: Jira Issue OCPBUGS-38290: All pull requests linked via external trackers have merged: Jira Issue OCPBUGS-38290 has been moved to the MODIFIED state. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
@anik120: new pull request created: #840 In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
[ART PR BUILD NOTIFIER] Distgit: operator-lifecycle-manager |
|
[ART PR BUILD NOTIFIER] Distgit: ose-operator-framework-tools |
|
[ART PR BUILD NOTIFIER] Distgit: operator-registry |
Using "available CatalogSources" information from the registry-client cache was causing cache inconsistency problems.
This has showed up multiple times in production environments over the years, manifesting itself in the form of the all subscriptions in a namespace being transitioned into an error state when a Catalogsource that the cache claims to exist, has actually been deleted from the cluster, but the cache was not updated.
The Subscriptions are transitioned to an error state because of the deleted catalogsource with the follwing error message:
"message": "failed to populate resolver cache from source : failed to list
bundles: rpc error: code = Unavailable desc = connection error: desc = "transport:
Error while dialing dial tcp: lookup ..svc on 172.....: no such host"",
"reason": "ErrorPreventedResolution",
"status": "True",
"type": "ResolutionFailed"
This PR switches the information lookup from the cache, to using a client to list the CatalogSources present in the cluster.
Upstream-repository: operator-lifecycle-manager
Upstream-commit: ff9084a24b19848c02c2cd4b7d827e057f7f8b11