# PR 380: Manually Testing that ConfigMaps are No Longer Unexpectedly Updated By The ConfigMap Label Selector Bug

> I've tested your changes manually and was happy to see that the "random" (or rather, unexpected) updates to the target configmaps no longer take place.

I wrote this page as part of my review of the [PR 380](https://github.com/cert-manager/trust-manager/pull/380). This PR introduces a deterministic ordering to the CA bundle and aims to fix two things:

1. On one hand, a feature that was requested: people want to be able to change the order of the sources in the `sources` array without having the configmaps updated.
2. On the other hand, a serious bug that forces the target configmaps to be updated for no reason and seems to only occur when label selectors are used.

Item (2) doesn't have a clear minimal working example, so I decided to reproduce it here.

My hypotheses for item (2) are:

- H1: The API server doesn't return the results of "listing with label selectors" in a deterministic way,
- H2: The controller-runtime cache doesn't yield deterministic results when using label selectors.

Let's use the last version of trust-manager (v0.11.0):

Installation:

kind cluster create
helm upgrade -i cert-manager jetstack/cert-manager --create-namespace -n cert-manager --set crds.enabled=true
helm upgrade -i trust-manager -n cert-manager jetstack/trust-manager --set crds.enabled=true --set app.logLevel=5 --version v0.11.0

Now, I created a bundle using the label selector mechanism:

apiVersion: trust.cert-manager.io/v1alpha1
kind: Bundle
metadata:
  name: default-ca
spec:
  sources:
    - useDefaultCAs: true
    - configMap:
        selector:
          matchLabels:
            inject-to-default-ca: "true"
        key: "root-ca.pem"
  target:
    configMap:
      key: "root-certs.pem"
    additionalFormats:
      jks:
        key: "root-certs.jks"
      pkcs12:
        key: "root-certs.p12"

I started watching changes for th `default-ca` configmap using kubectl-watch-diff and `delta`:

kubectl watch-diff cm default-ca | delta --paging=never

And I created a large number of configmaps using this label using the following script:

#!/bin/bash
set -eo pipefail

# For loop that loops over each certificate from /etc/ssl/cert.pem using perl, not grep.
mkdir -p /tmp/csplit
csplit -k -f /tmp/csplit/ -s -n 3 /etc/ssl/cert.pem '/END CERTIFICATE/+1' '{1000}' || true
find /tmp/csplit -type f | while read -r cert; do
    if ! v=$(openssl x509 -in "$cert"); then
        continue
    fi
    echo "$v" >"$cert"
done

find /tmp/csplit -type f | while read -r cert; do
    kubectl create configmap -n cert-manager cacerts-"$(basename "$cert")" --from-file=root-ca.pem="$cert" --dry-run=client -o yaml | kubectl apply -f -
    kubectl label configmap -n cert-manager cacerts-"$(basename "$cert")" inject-to-default-ca=true --dry-run=client -o yaml | kubectl apply -f -
done

Now, wait a bit for the output of `kubectl watch-diff` to settle.

Once it has settled, let's trigger a reconciliation by restarting trust-manager every 10 seconds:

for : ; do k rs delete -n cert-manager -n app.kubernetes.io/name=trust-manager; sleep 10; done

(I delete the replicasets instead of using "rollout restart" because I had temporarily disabled the leader election and needed a single instance to be running at any time)

You should see once in a while the configmap being updated.

Now, if you move to the PR 380's branch:

make oci-push-manager \
  oci_manager_image_name=ttl.sh/mael/trust-manager \
  oci_manager_image_tag=pr380

Now, let's use that image:

helm upgrade -i trust-manager -n cert-manager jetstack/trust-manager --set crds.enabled=true \
  --set app.logLevel=5 \
  --set image.repository=ttl.sh/mael/trust-manager \
  --set image.tag=pr380 \
  --set image.pullPolicy=Always

If we restart trust-manager every now and then using the `for` loop from above, we no longer see changes in the output of `kubectl watch-diff`.

I haven't been able to go further and haven't found the root cause of the problem. I don't know if H1 or H2 is correct.