chore(test): Added CustomMatcher for asserting on k8 errors for better readability

[harishsurf]

Description of the change:
This PR is a result of discussion from here

Reviewer Checklist

• Implementation matches the proposed design, or proposal is updated to match implementation
• Sufficient unit test coverage
• Sufficient end-to-end test coverage
• Docs updated or added to /docs
• Commit messages sensible and descriptive

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: harishsurf
To complete the pull request process, please assign benluddy
You can assign the PR to them by writing /assign @benluddy in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[of-deploy-bot]

This PR failed 1 out of 1 times with 1 individual failed tests and 4 skipped tests. A test is considered flaky if failed on multiple commits.

totaltestcount: 1
failedtestcount: 1
flaketestcount: 1
skippedtestcount: 4
flaketests:

• classname: End-to-end
  name: 'Subscription creation if not installed'
  counts: 3
  details:

  • count: 3
    error: |4-

    /home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/subscription_e2e_test.go:51
    Timed out after 60.001s.
    failed to await deletion of test csvs
    Expected
        <[]v1alpha1.ClusterServiceVersion | len:1, cap:1>: [
            {
                TypeMeta: {
                    Kind: "ClusterServiceVersion",
                    APIVersion: "operators.coreos.com/v1alpha1",
                },
                ObjectMeta: {
                    Name: "myapp-beta",
                    GenerateName: "",
                    Namespace: "operators",
                    SelfLink: "/apis/operators.coreos.com/v1alpha1/namespaces/operators/clusterserviceversions/myapp-beta",
                    UID: "620ff730-7a40-473c-8ceb-26c1bdef25dd",
                    ResourceVersion: "2777",
                    Generation: 1,
                    CreationTimestamp: {
                        Time: 2020-08-20T14:07:04Z,
                    },
                    DeletionTimestamp: nil,
                    DeletionGracePeriodSeconds: nil,
                    Labels: nil,
                    Annotations: {
                        "olm.operatorNamespace": "operators",
                        "olm.targetNamespaces": "",
                        "olm.operatorGroup": "global-operators",
                    },
                    OwnerReferences: nil,
                    Finalizers: nil,
                    ClusterName: "",
                    ManagedFields: nil,
                },
                Spec: {
                    InstallStrategy: {
                        StrategyName: "deployment",
                        StrategySpec: {
                            DeploymentSpecs: [
                                {
                                    Name: "dep-gppjq",
                                    Spec: {
                                        Replicas: 1,
                                        Selector: {
                                            MatchLabels: {...: ...},
                                            MatchExpressions: nil,
                                        },
                                        Template: {
                                            ObjectMeta: {
                                                Name: ...,
                                                GenerateName: ...,
                                                Namespace: ...,
                                                SelfLink: ...,
                                                UID: ...,
                                                ResourceVersion: ...,
                                                Generation: ...,
                                                CreationTimestamp: ...,
                                                DeletionTimestamp: ...,
                                                DeletionGracePeriodSeconds: ...,
                                                Labels: ...,
                                                Annotations: ...,
                                                OwnerReferences: ...,
                                                Finalizers: ...,
                                                ClusterName: ...,
                                                ManagedFields: ...,
                                            },
                                            Spec: {
                                                Volumes: ...,
                                                InitContainers: ...,
                                                Containers: ...,
                                                EphemeralContainers: ...,
                                                RestartPolicy: ...,
                                                TerminationGracePeriodSeconds: ...,
                                                ActiveDeadlineSeconds: ...,
                                                DNSPolicy: ...,
                                                NodeSelector: ...,
                                                ServiceAccountName: ...,
                                                DeprecatedServiceAccount: ...,
                                                AutomountServiceAccountToken: ...,
                                                NodeName: ...,
                                                HostNetwork: ...,
                                                HostPID: ...,
                                                HostIPC: ...,
                                                ShareProcessNamespace: ...,
                                                SecurityContext: ...,
                                                ImagePullSecrets: ...,
                                                Hostname: ...,
                                                Subdomain: ...,
                                                Affinity: ...,
                                                SchedulerName: ...,
                                                Tolerations: ...,
                                                HostAliases: ...,
                                                PriorityClassName: ...,
                                                Priority: ...,
                                                DNSConfig: ...,
                                                ReadinessGates: ...,
                                                RuntimeClassName: ...,
                                                EnableServiceLinks: ...,
                                                PreemptionPolicy: ...,
                                                Overhead: ...,
                                                TopologySpreadConstraints: ...,
                                            },
                                        },
                                        Strategy: {Type: "", RollingUpdate: nil},
                                        MinReadySeconds: 0,
                                        RevisionHistoryLimit: nil,
                                        Paused: false,
                                        ProgressDeadlineSeconds: nil,
                                    },
                                    Label: nil,
                                },
                            ],
                            Permissions: nil,
                            ClusterPermissions: nil,
                        },
                    },
                    Version: {
                        Version: {Major: 0, Minor: 1, Patch: 1, Pre: nil, Build: nil},
                    },
                    Maturity: "",
                    CustomResourceDefinitions: {Owned: nil, Required: nil},
                    APIServiceDefinitions: {Owned: nil, Required: nil},
                    WebhookDefinitions: nil,
                    NativeAPIs: nil,
                    MinKubeVersion: "",
                    DisplayName: "",
                    Description: "",
                    Keywords: nil,
                    Maintainers: nil,
                    Provider: {Name: "", URL: ""},
                    Links: nil,
                    Icon: nil,
                    InstallModes: [
                        {Type: "OwnNamespace", Supported: true},
                        {
                            Type: "SingleNamespace",
                            Supported: true,
                        },
                        {
                            Type: "MultiNamespace",
                            Supported: true,
                        },
                        {
                            Type: "AllNamespaces",
                            Supported: true,
                        },
                    ],
                    Replaces: "myapp-stable",
                    Labels: nil,
                    Annotations: nil,
                    Selector: nil,
                },
                Status: {
                    Phase: "Succeeded",
                    Message: "install strategy completed with no errors",
                    Reason: "InstallSucceeded",
                    LastUpdateTime: {
                        Time: 2020-08-20T14:07:06Z,
                    },
                    LastTransitionTime: {
                        Time: 2020-08-20T14:07:06Z,
                    },
                    Conditions: [
                        {
                            Phase: "Pending",
                            Message: "requirements not yet checked",
                            Reason: "RequirementsUnknown",
                            LastUpdateTime: {
                                Time: 2020-08-20T14:07:04Z,
                            },
                            LastTransitionTime: {
                                Time: 2020-08-20T14:07:04Z,
                            },
                        },
                        {
                            Phase: "InstallReady",
                            Message: "all requirements found, attempting install",
                            Reason: "AllRequirementsMet",
                            LastUpdateTime: {
                                Time: 2020-08-20T14:07:04Z,
                            },
                            LastTransitionTime: {
                                Time: 2020-08-20T14:07:04Z,
                            },
                        },
                        {
                            Phase: "Installing",
                            Message: "waiting for install components to report healthy",
                            Reason: "InstallSucceeded",
                            LastUpdateTime: {
                                Time: 2020-08-20T14:07:05Z,
                            },
                            LastTransitionTime: {
                                Time: 2020-08-20T14:07:05Z,
                            },
                        },
                        {
                            Phase: "Installing",
                            Message: "installing: waiting for deployment dep-gppjq to become ready: Waiting for rollout to finish: 0 of 1 updated replicas are available...\n",
                            Reason: "InstallWaiting",
                            LastUpdateTime: {
                                Time: 2020-08-20T14:07:05Z,
                            },
                            LastTransitionTime: {
                                Time: 2020-08-20T14:07:05Z,
                            },
                        },
                        {
                            Phase: "Succeeded",
                            Message: "install strategy completed with no errors",
                            Reason: "InstallSucceeded",
                            LastUpdateTime: {
                                Time: 2020-08-20T14:07:06Z,
                            },
                            LastTransitionTime: {
                                Time: 2020-08-20T14:07:06Z,
                            },
                        },
                    ],
                    RequirementStatus: nil,
                    CertsLastUpdated: nil,
                    CertsRotateAt: nil,
                },
            },
        ]
    to be empty
    /home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/util_test.go:496
    

  meandurationsec: 129.45367533333336
  skippedtests:
• classname: End-to-end
  name: 'Subscriptions create required objects from Catalogs Given a Namespace
  when a CatalogSource is created with a bundle that contains prometheus objects
  creating a subscription using the CatalogSource should install the operator
  successfully'
  counts: 1
  details: []
  meandurationsec: 22.246817
• classname: End-to-end
  name: 'Subscription updates existing install plan'
  counts: 1
  details: []
  meandurationsec: 0.586992
• classname: End-to-end
  name: 'Subscriptions create required objects from Catalogs Given a Namespace
  when a CatalogSource is created with a bundle that contains prometheus objects
  creating a subscription using the CatalogSource should have created the expected
  prometheus objects'
  counts: 1
  details: []
  meandurationsec: 12.141979
• classname: End-to-end
  name: 'Catalog image update'
  counts: 1
  details: []
  meandurationsec: 0.0528

[benluddy]

/hold

[benluddy]

The string equality matching here is pretty confusing, and it sort of reverses the way Gomega assertions are written -- that is, the argument to Expect is the test value on which the assertion is being made, and the parameter to Should is the matcher.

I was imagining a custom GomegaMatcher implementation, so test authors could write an assertion something like Expect(err).To(SatisfyErrorPredicate(errors.IsNotFound)) instead of Expect(errors.IsNotFound(err)).To(BeTrue()), and get more useful test failure output than Expected: true Actual: false. It could live in the test/e2e/dsl package, too.

[benluddy]

https://golang.org/pkg/runtime/#FuncForPC can also return nil, which would cause a panic and force users to stop and understand the implementation of this function. Magic like reflection needs to be extra robust because it's so much harder to understand when it breaks.

[benluddy]

Closing this PR as it's idle and won't be merged as-is. I'm happy to discuss testing quality-of-life improvements with you any time if you're interested in picking it back up.