OCPBUGS-5523: Catalog, fatal error: concurrent map read and map write

[dtfranz]

Problem:
The metrics pkg uses a shared map var to keep a record of Prometheus counters from Subscription objects, which is not protected from concurrent read/write access. This can result in container restarts.

Solution:
This PR adds a mutex to lock and unlock before and after each map access instance. Any number of requests to read from the map may happen concurrently, but when a write operation occurs nothing else can access the map until the write is finished.

Motivation for the change:

OCPBUGS-5523

[openshift-ci-robot]

@dtfranz: This pull request references Jira Issue OCPBUGS-5523, which is invalid:

• expected the bug to be in one of the following states: NEW, ASSIGNED, POST, but it is ON_QA instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

Problem:
The metrics pkg uses a shared map var to keep a record of Prometheus counters from Subscription objects, which is not protected from concurrent read/write access. This can result in container restarts.

Solution:
This PR adds a mutex to lock and unlock before and after each map access instance. Any number of requests to read from the map may happen concurrently, but when a write operation occurs nothing else can access the map until the write is finished.

Motivation for the change:

OCPBUGS-5523

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[dtfranz]

/cherry-pick 4.10

[openshift-cherrypick-robot]

@dtfranz: once the present PR merges, I will cherry-pick it on top of 4.10 in a new PR and assign it to you.

In response to this:

/cherry-pick 4.10

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[awgreene]

/approve
/retest

[dtfranz]

/test e2e-gcp-ovn

[dtfranz]

/retest

[awgreene]

@dtfranz seems like a gcp failure:

{  failed to acquire lease for "gcp-quota-slice": resources not found}

[awgreene]

/retest

[openshift-ci]

@dtfranz: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[dtfranz]

/jira refresh

[openshift-ci-robot]

@dtfranz: This pull request references Jira Issue OCPBUGS-5523, which is invalid:

• expected the bug to be in one of the following states: NEW, ASSIGNED, POST, but it is Verified instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[dtfranz]

/jira refresh

[openshift-ci-robot]

@dtfranz: This pull request references Jira Issue OCPBUGS-5523, which is invalid:

• expected the bug to be in one of the following states: NEW, ASSIGNED, POST, but it is Verified instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[tmshort]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: awgreene, dtfranz, tmshort

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [awgreene]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[dtfranz]

/jira refresh

[openshift-ci-robot]

@dtfranz: This pull request references Jira Issue OCPBUGS-5523, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.13.0) matches configured target version for branch (4.13.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

No GitHub users were found matching the public email listed for the QA contact in Jira ([email protected]), skipping review request.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci-robot]

@dtfranz: All pull requests linked via external trackers have merged:

• operator-framework/operator-lifecycle-manager#2913
• openshift/operator-framework-olm#429

Jira Issue OCPBUGS-5523 has been moved to the MODIFIED state.

In response to this:

Problem:
The metrics pkg uses a shared map var to keep a record of Prometheus counters from Subscription objects, which is not protected from concurrent read/write access. This can result in container restarts.

Solution:
This PR adds a mutex to lock and unlock before and after each map access instance. Any number of requests to read from the map may happen concurrently, but when a write operation occurs nothing else can access the map until the write is finished.

Motivation for the change:

OCPBUGS-5523

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-cherrypick-robot]

@dtfranz: cannot checkout 4.10: error checking out 4.10: exit status 1. output: error: pathspec '4.10' did not match any file(s) known to git

In response to this:

/cherry-pick 4.10

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[dtfranz]

/cherry-pick release-4.10

[openshift-cherrypick-robot]

@dtfranz: #429 failed to apply on top of branch "release-4.10":

Applying: OCPBUGS-5523: Catalog, fatal error: concurrent map read and map write (#2913)
Using index info to reconstruct a base tree...
M	staging/operator-lifecycle-manager/pkg/metrics/metrics.go
Falling back to patching base and 3-way merge...
Auto-merging staging/operator-lifecycle-manager/pkg/metrics/metrics.go
CONFLICT (content): Merge conflict in staging/operator-lifecycle-manager/pkg/metrics/metrics.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 OCPBUGS-5523: Catalog, fatal error: concurrent map read and map write (#2913)
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick release-4.10

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[awgreene]

/cherry-pick release-4.12

[openshift-cherrypick-robot]

@awgreene: new pull request created: #436

In response to this:

/cherry-pick release-4.12

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.