---
# System prepended metadata

title: diving into olm e2e catalog failures
tags: [test-debugging]

---

---
title: diving into olm e2e catalog failures
authors:
  - "@grokspawn"
reviewers:
  - TBD
approvers:
  - TBD
tags: test-debugging
---


###### tags: test-debugging

# diving into olm e2e catalog failures

original thread: https://coreos.slack.com/archives/G3T7N42NP/p1666358415190779

jordank
 Today at 8:20 AM
languishing non-deppy PRs:
api OWNER changes: https://github.com/operator-framework/api/pull/268
api absorbing declcfg to make it easier for folks to build pieces that “speak FBC”: https://github.com/operator-framework/api/pull/266
In addition, I’d like to write a de-flake e2e-gcp-olm story on the OLM component to resolve its many, many, many failures and to /override e2e-gcp-olm for this PR
42 replies


tim
:spiral_calendar_pad:  1 hour ago
wait... this is master branch. we should be having green CI runs on that branch. seems like we either regressed in the past release or so, or catherine has terrible luck.


tim
:spiral_calendar_pad:  1 hour ago
have we dive into those e2e failures? from a glance, it looks like the same installplan tests have been consistency failing.


tim
:spiral_calendar_pad:  1 hour ago
Using the /logs/artifacts/install-plan-e2e-px5gj output directory
Storing the test artifact output in the /logs/artifacts/install-plan-e2e-px5gj directory
Collecting get catalogsources -o yaml output...
./collect-ci-artifacts.sh: line 29: oc: command not found
failed to collect namespace artifacts: exit status 127
I'll throw up a PR now to fix this...


jordank
  1 hour ago
I did, to my limits: https://coreos.slack.com/archives/G3T7N42NP/p1666274571614449



> jordank
> I need a lease-an-eyeball service, with my propensity to ask for them, but anyway …
> need some help with a persistent downstream e2e failure for this PR , failing here .  My MO would be to > extract the steps and replay them manually, but that’s where I run into my ignorance.
> e2e-gcp-olm has failed every time for this PR, but if I check previous PRs'  results, this fails far 
> more often than it succeeds, but it eventually does after a handful of tries.   For contrast, this test 
> has been executed (and fails) 17 times in the current PR.
>Thread in team-olm | Yesterday at 9:02 AM | View message


jordank
  1 hour ago
Using the /logs/artifacts/install-plan-e2e-px5gj output directory
Storing the test artifact output in the /logs/artifacts/install-plan-e2e-px5gj directory
Collecting get catalogsources -o yaml output...
./collect-ci-artifacts.sh: line 29: oc: command not found
failed to collect namespace artifacts: exit status 127
Where did you find this?
I don’t see it anywhere in the prow data (https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_operator-fr[…]rator-framework-olm-master-e2e-gcp-olm/1583156312722640896).


tim
:spiral_calendar_pad:  1 hour ago
https://github.com/operator-framework/operator-lifecycle-manager/pull/2876 PR for fixing this


tim
:spiral_calendar_pad:  1 hour ago
I saw it in the e2e logs


tim
:spiral_calendar_pad:  1 hour ago
tl;dr the oc binary exists downstream and present in $PATH in the CI environment, but when spawning the new process to run the CI artifacts script, we override the $PATH value when we need to be retaining it.


tim
:spiral_calendar_pad:  1 hour ago
have we tried running the downstream OLM e2e testing suite locally to figure out whether this is simply a flake?


tim
:spiral_calendar_pad:  1 hour ago
because this looks unrelated looking at the changes in the PR


jordank
  1 hour ago
I ran it locally yesterday. This failure did not repeat, but there were others.  I was focused on this test only.


tim
:spiral_calendar_pad:  1 hour ago
oof i thought we fixed the test pollution issue with the custom client


jordank
  1 hour ago
It’s totally unrelated to the PR changes.


tim
:spiral_calendar_pad:  1 hour ago
(assuming this is related to tests polluting unrelated tests due to poor cleanup)


tim
:spiral_calendar_pad:  1 hour ago
it looks like we're failing in the BeforeEach clause which is weird: https://github.com/openshift/operator-framework-olm/blob/master/staging/operator-lifecycle-manager/test/e2e/installplan_e2e_test.go#L161-L164


jordank
  1 hour ago
Yep.  Expects to see the installplan complete… and it was timing out trying to collect the namespace artifacts (I read as “trying to find out if we got there”).


tim
:spiral_calendar_pad:  1 hour ago
when an individual test case fails, we try to collect the relevant OLM testing artifacts to improve debuggability in this situation. so in this context, both the individual job failed and we silently failed to collect the relevant debug artifacts.


tim
:spiral_calendar_pad:  1 hour ago
I threw up a PR that should fix the latter that we'll need to port downstream. (edited) 


tim
:spiral_calendar_pad:  1 hour ago
https://kubernetes.io/docs/reference/using-api/deprecation-guide/#poddisruptionbudget-v125 is no longer served in 1.25 and kube 1.25 just landed downstream
KubernetesKubernetes
Deprecated API Migration Guide
As the Kubernetes API evolves, APIs are periodically reorganized or upgraded. When APIs evolve, the old API is deprecated and eventually removed. This page contains information you need to know when migrating from deprecated API versions to newer and more stable API versions. Removed APIs by release v1.27 The v1.27 release will stop serving the following deprecated API versions: CSIStorageCapacity The storage.k8s.io/v1beta1 API version of CSIStorageCapacity will no longer be served in v1.


tim
:spiral_calendar_pad:  1 hour ago
```log
time="2022-10-19T18:27:56Z" level=info msg="skip processing installplan without status - subscription sync responsible for initial status" id=s8ZOy ip=test-plan-api namespace=deprecated-e2e-bb8hb phase=
time="2022-10-19T18:27:56Z" level=info msg=syncing id=h780a ip=test-plan-api namespace=deprecated-e2e-bb8hb phase=Installing
time="2022-10-19T18:27:56Z" level=info msg="could not query for GVK in api discovery" err="the server could not find the requested resource" group=verticalpodautoscalers.autoscaling.k8s.io kind=VerticalPodAutoscaler version=v1
E1019 18:27:56.748459       1 queueinformer_operator.go:298] sync {"update" "deprecated-e2e-bb8hb/test-plan-api"} failed: api-server resource not found installing VerticalPodAutoscaler my.thing: GroupVersionKind verticalpodautoscalers.autoscaling.k8s.io/v1, Kind=VerticalPodAutoscaler not found on the cluster. This API may have been deprecated and removed, see https://kubernetes.io/docs/reference/using-api/deprecation-guide/ for more information.
time="2022-10-19T18:27:56Z" level=info msg=syncing id=mt2GN ip=test-plan-api namespace=deprecated-e2e-bb8hb phase=Installing
```

tim
:spiral_calendar_pad:  1 hour ago
seems like a brittle test here


jordank
  1 hour ago
Where did you find that log?


tim
:spiral_calendar_pad:  1 hour ago
https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_oper[…]-operator-7b998d9968-89rg6_catalog-operator.log
tim
:spiral_calendar_pad:  1 hour ago
if you're asking how I found that log:
- clicked on the failed e2e test case dropdown and pressed the "open stderr" window
- looked for the namespace this resource was deployed in and found the "tearing down the install-plan-e2e-9w7zt namespace" message
- went back to the original window, and clicked the "artifacts" tab in the top right
- navigated to the artifacts -> e2e-gcp-olm -> gather-extra -> artifacts -> pods directory hierarchy and searched for the catalog-operators logs
- ctrl+f for "install-plan-e2e-9w7zt" namespace to find any relevant logs
- saw the "sync {"update" "install-plan-e2e-9w7zt/test-plan-l6qzw"} failed: api-server resource not found installing PodDisruptionBudget test-pdb-47rhz: GroupVersionKind /policy/v1beta1, Kind=PodDisruptionBudget not found on the cluster. This API may have been deprecated and removed, see https://kubernetes.io/docs/reference/using-api/deprecation-guide/ for more information." log



