https://issues.redhat.com/browse/OCPBUGS-35210
related "waiting on customer" case: https://access.redhat.com/support/cases/#/case/03819995

I wanted to take a scan through available m-gs in the last open attached case.

omc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.13.45   True        False         2d      Cluster version is 4.13.45

**looks like the attempted upgrade was v5.7.11 --> v5.8.10**
this is where the issue was noticed. the subscription actually sets startingCSV to `elasticsearch-operator.5.5.12`

from m-g dated 5 Aug 2024 12:35
attempt to upgrade the elasticsearch-operator

brute grep

ag -il -- elasticsearch-operator-metrics-token .
must-gather.local.4105022038456865091/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-1580e89084e76f12e73a75fd3a0fe88c8974ea444d684ac0c8f275b04b67c9c0/namespaces/openshift-operator-lifecycle-manager/pods/catalog-operator-b9d8b7d48-dczm5/catalog-operator/catalog-operator/logs/current.log
must-gather.local.4105022038456865091/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-1580e89084e76f12e73a75fd3a0fe88c8974ea444d684ac0c8f275b04b67c9c0/namespaces/openshift-marketplace/pods/9fa29fbc929a827c4227a104a5d77f85cb7c1f59be512fa15b6f25d056mcls5/extract/extract/logs/current.log
must-gather.local.4105022038456865091/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-1580e89084e76f12e73a75fd3a0fe88c8974ea444d684ac0c8f275b04b67c9c0/namespaces/openshift-marketplace/core/configmaps.yaml
must-gather.local.4105022038456865091/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-1580e89084e76f12e73a75fd3a0fe88c8974ea444d684ac0c8f275b04b67c9c0/namespaces/openshift-monitoring/pods/prometheus-operator-7dd89b645b-jn8ck/prometheus-operator/prometheus-operator/logs/current.log
must-gather.local.4105022038456865091/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-1580e89084e76f12e73a75fd3a0fe88c8974ea444d684ac0c8f275b04b67c9c0/namespaces/openshift-operators-redhat/operators.coreos.com/installplans/install-g7lmh.yaml
must-gather.local.4105022038456865091/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-1580e89084e76f12e73a75fd3a0fe88c8974ea444d684ac0c8f275b04b67c9c0/namespaces/openshift-operators-redhat/monitoring.coreos.com/servicemonitors/elasticsearch-operator-metrics-monitor.yaml

file `must-gather.local.4105022038456865091/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-1580e89084e76f12e73a75fd3a0fe88c8974ea444d684ac0c8f275b04b67c9c0/namespaces/openshift-monitoring/pods/prometheus-operator-7dd89b645b-jn8ck/prometheus-operator/prometheus-operator/logs/current.log` shows an error accessing the missing token:

2024-08-02T22:07:48.440246695Z level=warn ts=2024-08-02T22:07:48.440187077Z caller=operator.go:1917 component=prometheusoperator msg="skipping servicemonitor" error="failed to get CA: unable to get secret \"elasticsearch-operator-metrics-token\": secrets \"elasticsearch-operator-metrics-token\" not found" servicemonitor=openshift-operators-redhat/elasticsearch-operator-metrics-monitor namespace=openshift-monitoring prometheus=k8s

logs on installplan `install-g7lmh` to resolve `elastiserarch-operator-metrics-token` for bundle version `elasticsearch-operator.v5.8.10`

`must-gather.local.4105022038456865091/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-1580e89084e76f12e73a75fd3a0fe88c8974ea444d684ac0c8f275b04b67c9c0/namespaces/openshift-marketplace/core/configmaps.yaml` defines `elasticsearch-operator-metrics-token_v1_secret.yaml` which is a doubly base64-encoded gzip archive that looks like

apiVersion: v1
kind: Secret
metadata:
  annotations:
    kubernetes.io/service-account.name: elasticsearch-operator-metrics-reader
  labels:
    name: elasticsearch-operator
  name: elasticsearch-operator-metrics-token
type: kubernetes.io/service-account-token

omc get csv -A | grep -i elasticsearch0operator | wc -l 

shows 169 instances of the operator, all at v5.8.10

need to spin a cluster up tomorrow to see if I can reproduce, which should be the case if this is an operator issue
(edit: spun 4.13 OCP cluster and upgraded the elasticsearch-operator through the whole range of channels, but did not reproduce the issue)

subscription check
`omc get subscription -n openshift-operators-redhat elasticsearch-operator -o yaml`

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  creationTimestamp: "2023-07-07T23:32:09Z"
  generation: 4
  labels:
    operators.coreos.com/elasticsearch-operator.openshift-operators-redhat: ""
  name: elasticsearch-operator
  namespace: openshift-operators-redhat
  resourceVersion: "1701815582"
  uid: 618d7ab0-615c-4e3d-9dc3-dc3148597cda
spec:
  channel: stable-5.8
  installPlanApproval: Manual
  name: elasticsearch-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
  startingCSV: elasticsearch-operator.5.5.12
status:
  catalogHealth:
  - catalogSourceRef:
      apiVersion: operators.coreos.com/v1alpha1
      kind: CatalogSource
      name: certified-operators
      namespace: openshift-marketplace
      resourceVersion: "1701693875"
      uid: 1a8c069a-afa8-4d33-a762-f56ea3295036
    healthy: true
    lastUpdated: "2024-08-02T19:06:23Z"
  - catalogSourceRef:
      apiVersion: operators.coreos.com/v1alpha1
      kind: CatalogSource
      name: community-operators
      namespace: openshift-marketplace
      resourceVersion: "1701693836"
      uid: 786263c8-0110-4fee-8edc-a3a01d1f6488
    healthy: true
    lastUpdated: "2024-08-02T19:06:23Z"
  - catalogSourceRef:
      apiVersion: operators.coreos.com/v1alpha1
      kind: CatalogSource
      name: redhat-marketplace
      namespace: openshift-marketplace
      resourceVersion: "1701693881"
      uid: 00506496-e9b0-4111-87e2-6e4aa1d4a8a5
    healthy: true
    lastUpdated: "2024-08-02T19:06:23Z"
  - catalogSourceRef:
      apiVersion: operators.coreos.com/v1alpha1
      kind: CatalogSource
      name: redhat-operators
      namespace: openshift-marketplace
      resourceVersion: "1701693825"
      uid: 809a5e46-ed63-4647-b70a-92e57fce4f1c
    healthy: true
    lastUpdated: "2024-08-02T19:06:23Z"
  conditions:
  - lastTransitionTime: "2024-08-02T19:06:23Z"
    message: all available catalogsources are healthy
    reason: AllCatalogSourcesHealthy
    status: "False"
    type: CatalogSourcesUnhealthy
  currentCSV: elasticsearch-operator.v5.8.10
  installPlanGeneration: 7
  installPlanRef:
    apiVersion: operators.coreos.com/v1alpha1
    kind: InstallPlan
    name: install-g7lmh
    namespace: openshift-operators-redhat
    resourceVersion: "1701802260"
    uid: 61829d56-b0eb-43d0-8ffa-b756082fec51
  installedCSV: elasticsearch-operator.v5.8.10
  installplan:
    apiVersion: operators.coreos.com/v1alpha1
    kind: InstallPlan
    name: install-g7lmh
    uuid: 61829d56-b0eb-43d0-8ffa-b756082fec51
  lastUpdated: "2024-08-02T21:26:56Z"
  state: AtLatestKnown