# OpenShift 4.11 Certificate Rotation

## Aggregated API Servers

### CA Secret

~~~sh
oc -n openshift-kube-apiserver-operator get secret aggregator-client-signer -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
~~~

~~~sh
notBefore=Sep 29 12:52:23 2022 GMT
notAfter=Oct 29 12:52:24 2022 GMT
~~~

### Client Cert

~~~sh
oc -n openshift-kube-apiserver get secret aggregator-client -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
~~~

~~~sh
notBefore=Sep 29 13:08:34 2022 GMT
notAfter=Oct 29 12:52:24 2022 GMT
~~~

### CA Rotate

~~~sh
oc -n openshift-kube-apiserver-operator delete secret aggregator-client-signer
~~~

We can see the new validity:

~~~sh
oc -n openshift-kube-apiserver-operator get secret aggregator-client-signer -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
~~~

~~~sh
notBefore=Oct 4 16:58:33 2022 GMT
notAfter=Nov 3 16:58:34 2022 GMT
~~~

At this point, we need to rotate the certificates issued by the old CA, otherwise the trust chain will be broken:

~~~sh
oc -n openshift-kube-apiserver-operator get secret aggregator-client-signer -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/aggapica.crt

oc -n openshift-kube-apiserver get secret aggregator-client -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/aggapiclient.crt
~~~

~~~sh
openssl verify -verbose -CAfile /tmp/aggapica.crt /tmp/aggapiclient.crt
~~~

~~~sh
CN = system:openshift-aggregator
error 20 at 0 depth lookup: unable to get local issuer certificate
error /tmp/aggapiclient.crt: verification failed
~~~

Even if the trust fails with the new signer cert, the old signer cert CA is still available in the CABundle so services using old certs won't stop working.

~~~sh
oc -n openshift-kube-apiserver get cm aggregator-client-ca -o jsonpath='{.data.ca-bundle\.crt}' > /tmp/aggregator-cabundle.crt
~~~

~~~sh
openssl verify -verbose -CAfile /tmp/aggregator-cabundle.crt /tmp/aggapiclient.crt
~~~

~~~sh
/tmp/aggapiclient.crt : OK
~~~

Let's force the rotation of the issued certs:

~~~sh
oc -n openshift-kube-apiserver delete secret aggregator-client
~~~

~~~sh
oc -n openshift-kube-apiserver get secret aggregator-client -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
~~~

~~~sh
notBefore=Oct 4 16:59:36 2022 GMT
notAfter=Nov 3 16:58:34 2022 GMT
~~~

~~~sh
oc -n openshift-kube-apiserver get secret aggregator-client -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/aggapiclient.crt
~~~

~~~sh
openssl verify -verbose -CAfile /tmp/aggapica.crt /tmp/aggapiclient.crt
~~~

~~~sh
/tmp/aggapiclient.crt: OK
~~~

## MachineConfig Operator

Eventhough it's name Root CA, it's just used for generating the certificate used by the MachineConfigServer (openshift-machine-config-operator/machine-config-server-tls).

CA public key is present on disk at `/etc/kubernetes/ca.crt`.

The private key never reaches the cluster, is deleted during bootstrap.

Secret for the MCS (openshift-machine-config-operator/machine-config-server-tls) won't be recreated if deleted.

It cannot be rotated, an [RFE](https://issues.redhat.com/browse/RFE-3287) has been opened.

## Service Serving Certificates

The procedure for rotating the certs is already documented [here](https://docs.openshift.com/container-platform/4.10/security/certificates/service-serving-certificate.html#manually-rotate-service-ca_service-serving-certificate)

### CA Secret

~~~sh
oc -n openshift-service-ca get secret signing-key -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
~~~

~~~sh
notBefore=Sep 29 13:55:16 2022 GMT
notAfter=Nov 27 13:55:17 2024 GMT
~~~

### Client Certs

Any service in the cluster using the `service.beta.openshift.io/serving-cert-secret-name: <secret-name>`

~~~sh
oc -n openshift-kube-apiserver-operator get secret kube-apiserver-operator-serving-cert -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
~~~

~~~sh
notBefore=Sep 29 13:56:07 2022 GMT
notAfter=Sep 28 13:56:08 2024 GMT
~~~

### CA Rotate

~~~sh
oc -n openshift-service-ca delete secret signing-key
~~~

~~~sh
oc -n openshift-service-ca get secret signing-key -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
~~~

~~~sh
notBefore=Oct 4 17:00:50 2022 GMT
notAfter=Dec 2 17:00:51 2024 GMT
~~~

We need to wait for the certificates issued by the old CA to be automatically rotated, we can check the cluster operators:

~~~sh
oc get co
~~~

~~~sh
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
authentication 4.11.4 True True False 2s OAuthServerDeploymentProgressing: deployment/oauth-openshift.openshift-authentication: 1/3 pods have been updated to the latest generation
baremetal 4.11.4 True False False 8d
cloud-controller-manager 4.11.4 True False False 8d
cloud-credential 4.11.4 True False False 8d
cluster-autoscaler 4.11.4 True False False 8d
config-operator 4.11.4 True False False 8d
console 4.11.4 True False False 8d
csi-snapshot-controller 4.11.4 True False False 8d
dns 4.11.4 True False False 8d
etcd 4.11.4 True False False 8d
image-registry 4.11.4 True False False 8d
ingress 4.11.4 True False False 8d
insights 4.11.4 True False False 8d
kube-apiserver 4.11.4 True False False 8d
kube-controller-manager 4.11.4 True True False 8d NodeInstallerProgressing: 3 nodes are at revision 6; 0 nodes have achieved new revision 8
kube-scheduler 4.11.4 True True False 8d NodeInstallerProgressing: 3 nodes are at revision 7; 0 nodes have achieved new revision 8
kube-storage-version-migrator 4.11.4 True False False 8d
machine-api 4.11.4 True False False 8d
machine-approver 4.11.4 True False False 8d
machine-config 4.11.4 True False False 8d
marketplace 4.11.4 True False False 8d
monitoring 4.11.4 True False False 8d
network 4.11.4 True False False 8d
node-tuning 4.11.4 True False False 8d
openshift-apiserver 4.11.4 False False False 1s APIServicesAvailable: "image.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
openshift-controller-manager 4.11.4 True True False 8d Progressing: daemonset/controller-manager: updated number scheduled is 0, desired number scheduled is 3
openshift-samples 4.11.4 True False False 8d
operator-lifecycle-manager 4.11.4 True False False 8d
operator-lifecycle-manager-catalog 4.11.4 True False False 8d
operator-lifecycle-manager-packageserver 4.11.4 True False False 8d
service-ca 4.11.4 True False False 8d
storage 4.11.4 True False False 8d
~~~

After a while:

~~~sh
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
authentication 4.11.4 True False False 13m
baremetal 4.11.4 True False False 8d
cloud-controller-manager 4.11.4 True False False 8d
cloud-credential 4.11.4 True False False 8d
cluster-autoscaler 4.11.4 True False False 8d
config-operator 4.11.4 True False False 8d
console 4.11.4 True False False 8d
csi-snapshot-controller 4.11.4 True False False 8d
dns 4.11.4 True False False 8d
etcd 4.11.4 True False False 8d
image-registry 4.11.4 True False False 8d
ingress 4.11.4 True False False 8d
insights 4.11.4 True False False 8d
kube-apiserver 4.11.4 True False False 8d
kube-controller-manager 4.11.4 True False False 8d
kube-scheduler 4.11.4 True False False 8d
kube-storage-version-migrator 4.11.4 True False False 8d
machine-api 4.11.4 True False False 8d
machine-approver 4.11.4 True False False 8d
machine-config 4.11.4 True False False 8d
marketplace 4.11.4 True False False 8d
monitoring 4.11.4 True False False 8d
network 4.11.4 True False False 8d
node-tuning 4.11.4 True False False 8d
openshift-apiserver 4.11.4 True False False 13m
openshift-controller-manager 4.11.4 True False False 8d
openshift-samples 4.11.4 True False False 8d
operator-lifecycle-manager 4.11.4 True False False 8d
operator-lifecycle-manager-catalog 4.11.4 True False False 8d
operator-lifecycle-manager-packageserver 4.11.4 True False False 8d
service-ca 4.11.4 True False False 8d
storage 4.11.4 True False False 8d
~~~

~~~sh
oc -n openshift-kube-apiserver-operator get secret kube-apiserver-operator-serving-cert -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
~~~

~~~sh
notBefore=Oct 4 17:01:27 2022 GMT
notAfter=Oct 3 17:01:28 2024 GMT
~~~

The certificates have been renewed. Platform pods will be restarted, app pods may need to be restarted automatically to get the latest certificates.

~~~sh
oc -n openshift-service-ca get secret signing-key -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/serviceca.crt
oc -n openshift-kube-apiserver-operator get secret kube-apiserver-operator-serving-cert -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/servingcert.crt
~~~

~~~sh
openssl verify -verbose -CAfile /tmp/serviceca.crt /tmp/servingcert.crt
~~~

~~~sh
/tmp/servingcert.crt: OK
~~~

## ETCD Certificates

### CA Secret

~~~sh
oc -n openshift-config get secret etcd-signer -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
~~~

~~~sh
notBefore=Sep 21 07:50:31 2022 GMT
notAfter=Sep 18 07:50:32 2032 GMT
~~~

### Client Certs

~~~sh
for type in peer serving
do
for master in $(seq 0 2)
do
echo ""
echo etcd-$type-openshift-master-$master
oc -n openshift-etcd get secret etcd-$type-openshift-master-$master -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
done
done
~~~

~~~sh
etcd-peer-openshift-master-0
notBefore=Sep 21 08:10:38 2022 GMT
notAfter=Sep 20 08:10:39 2025 GMT

etcd-peer-openshift-master-1
notBefore=Sep 21 08:10:38 2022 GMT
notAfter=Sep 20 08:10:39 2025 GMT

etcd-peer-openshift-master-2
notBefore=Sep 21 08:12:58 2022 GMT
notAfter=Sep 20 08:12:59 2025 GMT

etcd-serving-openshift-master-0
notBefore=Sep 21 08:10:38 2022 GMT
notAfter=Sep 20 08:10:39 2025 GMT

etcd-serving-openshift-master-1
notBefore=Sep 21 08:10:38 2022 GMT
notAfter=Sep 20 08:10:39 2025 GMT

etcd-serving-openshift-master-2
notBefore=Sep 21 08:12:58 2022 GMT
notAfter=Sep 20 08:12:59 2025 GMT
~~~

~~~sh
for namespace in openshift-apiserver openshift-config openshift-etcd openshift-etcd-operator openshift-kube-apiserver openshift-oauth-apiserver
do
echo ""
echo "$namespace/etcd-client"
oc -n $namespace get secret etcd-client -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
done
~~~

### CA Rotate

~~~sh
oc -n openshift-config delete secret etcd-signer
~~~

At this point I'd expect the etcd-cluster-operator to recreate the cert but that's not the case. I opened an [RFE](https://issues.redhat.com/browse/RFE-3276) to get this fixed as it seems [it's not implemented](https://coreos.slack.com/archives/C027U68LP/p1660380716360259).

## ETCD Metrics Certificates

### CA Secret

~~~sh
oc -n openshift-config get secret etcd-metric-signer -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
~~~

~~~sh
notBefore=Sep 21 07:50:31 2022 GMT
notAfter=Sep 18 07:50:32 2032 GMT
~~~

### Client Certs

~~~sh
for master in $(seq 0 2)
do
echo ""
echo etcd-serving-metrics-openshift-master-$master
oc -n openshift-etcd get secret etcd-serving-metrics-openshift-master-$master -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
done
~~~

~~~sh
etcd-serving-metrics-openshift-master-0
notBefore=Sep 21 08:10:38 2022 GMT
notAfter=Sep 20 08:10:39 2025 GMT

etcd-serving-metrics-openshift-master-1
notBefore=Sep 21 08:10:39 2022 GMT
notAfter=Sep 20 08:10:40 2025 GMT

etcd-serving-metrics-openshift-master-2
notBefore=Sep 21 08:12:59 2022 GMT
notAfter=Sep 20 08:13:00 2025 GMT
~~~

### CA Rotate

~~~sh
oc -n openshift-config delete secret etcd-metric-signer
~~~

At this point I'd expect the etcd-cluster-operator to recreate the cert but that's not the case. I opened an [RFE](https://issues.redhat.com/browse/RFE-3276) to get this fixed as it seems [it's not implemented](https://coreos.slack.com/archives/C027U68LP/p1660380716360259).

## Kube API Server Client Certificates

### CA Secrets

~~~sh
echo "openshift-kube-apiserver-operator/kube-apiserver-to-kubelet-signer" && oc -n openshift-kube-apiserver-operator get secret kube-apiserver-to-kubelet-signer -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-kube-apiserver-operator/kube-control-plane-signer" && oc -n openshift-kube-apiserver-operator get secret kube-control-plane-signer -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-kube-apiserver-operator/node-system-admin-signer" && oc -n openshift-kube-apiserver-operator get secret node-system-admin-signer -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-kube-controller-manager-operator/csr-signer-signer" && oc -n openshift-kube-controller-manager-operator get secret csr-signer-signer -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-kube-controller-manager-operator/csr-signer" && oc -n openshift-kube-controller-manager-operator get secret csr-signer -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-kube-controller-manager/csr-signer" && oc -n openshift-kube-controller-manager get secret csr-signer -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
~~~

~~~sh
openshift-kube-apiserver-operator/kube-apiserver-to-kubelet-signer
notBefore=Sep 30 09:27:33 2022 GMT
notAfter=Sep 30 09:27:34 2023 GMT

openshift-kube-apiserver-operator/kube-control-plane-signer
notBefore=Sep 30 09:27:33 2022 GMT
notAfter=Nov 29 09:27:34 2022 GMT

openshift-kube-apiserver-operator/node-system-admin-signer
notBefore=Sep 30 09:27:34 2022 GMT
notAfter=Sep 30 09:27:35 2023 GMT

openshift-kube-controller-manager-operator/csr-signer-signer
notBefore=Sep 30 09:27:34 2022 GMT
notAfter=Nov 29 09:27:35 2022 GMT

openshift-kube-controller-manager-operator/csr-signer
notBefore=Sep 30 09:27:34 2022 GMT
notAfter=Oct 30 09:27:35 2022 GMT

openshift-kube-controller-manager/csr-signer
notBefore=Sep 30 09:27:34 2022 GMT
notAfter=Oct 30 09:27:35 2022 GMT
~~~

### Client Certs

~~~sh
echo "openshift-kube-apiserver-operator/node-system-admin-client" && oc -n openshift-kube-apiserver-operator get secret node-system-admin-client -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-kube-apiserver/control-plane-node-admin-client-cert-key" && oc -n openshift-kube-apiserver get secret control-plane-node-admin-client-cert-key -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-kube-apiserver/kubelet-client" && oc -n openshift-kube-apiserver get secret kubelet-client -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-kube-apiserver/check-endpoints-client-cert-key" && oc -n openshift-kube-apiserver get secret check-endpoints-client-cert-key -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-kube-controller-manager/kube-controller-manager-client-cert-key" && oc -n openshift-kube-controller-manager get secret kube-controller-manager-client-cert-key -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-config-managed/kube-controller-manager-client-cert-key" && oc -n openshift-config-managed get secret kube-controller-manager-client-cert-key -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-config-managed/kube-scheduler-client-cert-key" && oc -n openshift-config-managed get secret kube-scheduler-client-cert-key -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-kube-scheduler/kube-scheduler-client-cert-key" && oc -n openshift-kube-scheduler get secret kube-scheduler-client-cert-key -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-monitoring/metrics-client-certs" && oc -n openshift-monitoring get secret metrics-client-certs -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-oauth-apiserver/openshift-authenticator-certs" && oc -n openshift-oauth-apiserver get secret openshift-authenticator-certs -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
~~~

~~~sh
openshift-kube-apiserver-operator/node-system-admin-client
notBefore=Sep 30 09:50:33 2022 GMT
notAfter=Jan 28 09:50:34 2023 GMT

openshift-kube-apiserver/control-plane-node-admin-client-cert-key
notBefore=Sep 30 09:50:34 2022 GMT
notAfter=Oct 30 09:50:35 2022 GMT

openshift-kube-apiserver/kubelet-client
notBefore=Sep 30 09:50:34 2022 GMT
notAfter=Oct 30 09:50:35 2022 GMT

openshift-kube-apiserver/check-endpoints-client-cert-key
notBefore=Sep 30 09:50:34 2022 GMT
notAfter=Oct 30 09:50:35 2022 GMT

openshift-kube-controller-manager/kube-controller-manager-client-cert-key
notBefore=Sep 30 09:50:34 2022 GMT
notAfter=Oct 30 09:50:35 2022 GMT

openshift-config-managed/kube-controller-manager-client-cert-key
notBefore=Sep 30 09:50:34 2022 GMT
notAfter=Oct 30 09:50:35 2022 GMT

openshift-config-managed/kube-scheduler-client-cert-key
notBefore=Sep 30 09:50:34 2022 GMT
notAfter=Oct 30 09:50:35 2022 GMT

notBefore=Sep 30 09:50:34 2022 GMT
notAfter=Oct 30 09:50:35 2022 GMT

openshift-monitoring/metrics-client-certs
notBefore=Sep 30 09:45:35 2022 GMT
notAfter=Oct 30 09:27:35 2022 GMT

openshift-oauth-apiserver/openshift-authenticator-certs
notBefore=Sep 30 09:45:36 2022 GMT
notAfter=Oct 30 09:27:35 2022 GMT
~~~

### CA Rotate

~~~sh
oc -n openshift-kube-apiserver-operator delete secret kube-apiserver-to-kubelet-signer
oc -n openshift-kube-apiserver-operator delete secret kube-control-plane-signer
oc -n openshift-kube-apiserver-operator delete secret node-system-admin-signer
oc -n openshift-kube-controller-manager-operator delete secret csr-signer-signer
oc -n openshift-kube-controller-manager-operator delete secret csr-signer
oc -n openshift-kube-controller-manager delete secret csr-signer
~~~

~~~sh
echo "openshift-kube-apiserver-operator/kube-apiserver-to-kubelet-signer" && oc -n openshift-kube-apiserver-operator get secret kube-apiserver-to-kubelet-signer -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-kube-apiserver-operator/kube-control-plane-signer" && oc -n openshift-kube-apiserver-operator get secret kube-control-plane-signer -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-kube-apiserver-operator/node-system-admin-signer" && oc -n openshift-kube-apiserver-operator get secret node-system-admin-signer -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-kube-controller-manager-operator/csr-signer-signer" && oc -n openshift-kube-controller-manager-operator get secret csr-signer-signer -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-kube-controller-manager-operator/csr-signer" && oc -n openshift-kube-controller-manager-operator get secret csr-signer -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-kube-controller-manager/csr-signer" && oc -n openshift-kube-controller-manager get secret csr-signer -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
~~~

~~~sh
openshift-kube-apiserver-operator/kube-apiserver-to-kubelet-signer
notBefore=Oct 4 17:24:59 2022 GMT
notAfter=Oct 4 17:25:00 2023 GMT

openshift-kube-apiserver-operator/kube-control-plane-signer
notBefore=Oct 4 17:25:00 2022 GMT
notAfter=Dec 3 17:25:01 2022 GMT

openshift-kube-apiserver-operator/node-system-admin-signer
notBefore=Oct 4 17:25:00 2022 GMT
notAfter=Oct 4 17:25:01 2023 GMT

openshift-kube-controller-manager-operator/csr-signer-signer
notBefore=Oct 4 17:25:00 2022 GMT
notAfter=Dec 3 17:25:01 2022 GMT

openshift-kube-controller-manager-operator/csr-signer
notBefore=Oct 4 17:25:00 2022 GMT
notAfter=Nov 3 17:25:01 2022 GMT

openshift-kube-controller-manager/csr-signer
notBefore=Oct 4 17:25:00 2022 GMT
notAfter=Nov 3 17:25:01 2022 GMT
~~~

Cluster operators will roll out new revisions of the components. We need to wait until all cluster operators have stabilized.

At this point, we need to rotate the certificates issued by the old CA, otherwise the trust chain will be broken:

~~~sh
oc -n openshift-kube-apiserver-operator get secret kube-apiserver-to-kubelet-signer -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/kubeletsigner.crt
oc -n openshift-kube-apiserver get secret kubelet-client -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/kubeletclient.crt
~~~

~~~sh
openssl verify -verbose -CAfile /tmp/kubeletsigner.crt /tmp/kubeletclient.crt
~~~

~~~sh
O = kube-master, CN = system:kube-apiserver
error 20 at 0 depth lookup: unable to get local issuer certificate
error /tmp/kubeletclient.crt: verification failed
~~~

As we explained earlier, even if the trust is broken with the new CA, the old CA is still present in the bundle.

Let's force the rotation of the issued certs:

~~~sh
oc -n openshift-kube-apiserver-operator delete secret node-system-admin-client
oc -n openshift-kube-apiserver delete secret control-plane-node-admin-client-cert-key
oc -n openshift-kube-apiserver delete secret kubelet-client
oc -n openshift-kube-apiserver delete secret check-endpoints-client-cert-key
oc -n openshift-kube-controller-manager delete secret kube-controller-manager-client-cert-key
oc -n openshift-config-managed delete secret kube-controller-manager-client-cert-key
oc -n openshift-config-managed delete secret kube-scheduler-client-cert-key
oc -n openshift-kube-scheduler delete secret kube-scheduler-client-cert-key
oc -n openshift-monitoring delete secret metrics-client-certs
oc -n openshift-oauth-apiserver delete secret openshift-authenticator-certs
~~~

~~~sh
oc -n openshift-kube-apiserver get secret kubelet-client -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
~~~

~~~sh
notBefore=Oct 4 17:35:08 2022 GMT
notAfter=Nov 3 17:35:09 2022 GMT
~~~

~~~sh
oc -n openshift-kube-apiserver get secret kubelet-client -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/kubeletclient.crt
~~~

~~~sh
openssl verify -verbose -CAfile /tmp/kubeletsigner.crt /tmp/kubeletclient.crt
~~~

~~~sh
/tmp/kubeletclient.crt: OK
~~~

We can check new validity periods:

~~~sh
echo "openshift-kube-apiserver-operator/node-system-admin-client" && oc -n openshift-kube-apiserver-operator get secret node-system-admin-client -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-kube-apiserver/control-plane-node-admin-client-cert-key" && oc -n openshift-kube-apiserver get secret control-plane-node-admin-client-cert-key -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-kube-apiserver/kubelet-client" && oc -n openshift-kube-apiserver get secret kubelet-client -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-kube-apiserver/check-endpoints-client-cert-key" && oc -n openshift-kube-apiserver get secret check-endpoints-client-cert-key -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-kube-controller-manager/kube-controller-manager-client-cert-key" && oc -n openshift-kube-controller-manager get secret kube-controller-manager-client-cert-key -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-config-managed/kube-controller-manager-client-cert-key" && oc -n openshift-config-managed get secret kube-controller-manager-client-cert-key -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-config-managed/kube-scheduler-client-cert-key" && oc -n openshift-config-managed get secret kube-scheduler-client-cert-key -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-kube-scheduler/kube-scheduler-client-cert-key" && oc -n openshift-kube-scheduler get secret kube-scheduler-client-cert-key -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-monitoring/metrics-client-certs" && oc -n openshift-monitoring get secret metrics-client-certs -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-oauth-apiserver/openshift-authenticator-certs" && oc -n openshift-oauth-apiserver get secret openshift-authenticator-certs -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
~~~

~~~sh
openshift-kube-apiserver-operator/node-system-admin-client
notBefore=Oct 4 17:35:07 2022 GMT
notAfter=Feb 1 17:35:08 2023 GMT

openshift-kube-apiserver/control-plane-node-admin-client-cert-key
notBefore=Oct 4 17:35:07 2022 GMT
notAfter=Nov 3 17:35:08 2022 GMT

openshift-kube-apiserver/kubelet-client
notBefore=Oct 4 17:35:08 2022 GMT
notAfter=Nov 3 17:35:09 2022 GMT

openshift-kube-apiserver/check-endpoints-client-cert-key
notBefore=Oct 4 17:35:07 2022 GMT
notAfter=Nov 3 17:35:08 2022 GMT

openshift-kube-controller-manager/kube-controller-manager-client-cert-key
notBefore=Oct 4 17:35:07 2022 GMT
notAfter=Nov 3 17:35:08 2022 GMT

openshift-config-managed/kube-controller-manager-client-cert-key
notBefore=Oct 4 17:35:07 2022 GMT
notAfter=Nov 3 17:35:08 2022 GMT

openshift-config-managed/kube-scheduler-client-cert-key
notBefore=Oct 4 17:35:07 2022 GMT
notAfter=Nov 3 17:35:08 2022 GMT

openshift-kube-scheduler/kube-scheduler-client-cert-key
notBefore=Oct 4 17:35:07 2022 GMT
notAfter=Nov 3 17:35:08 2022 GMT

openshift-monitoring/metrics-client-certs
notBefore=Oct 4 17:30:09 2022 GMT
notAfter=Nov 3 17:25:01 2022 GMT

openshift-oauth-apiserver/openshift-authenticator-certs
notBefore=Oct 4 17:30:09 2022 GMT
notAfter=Nov 3 17:25:01 2022 GMT
~~~

Cluster operators will roll out new revisions of the components. We need to wait until all cluster operators have stabilized.

## Kube API Server Serving Certificates

### CA Secrets

~~~sh
echo "openshift-kube-apiserver-operator/loadbalancer-serving-signer" && oc -n openshift-kube-apiserver-operator get secret loadbalancer-serving-signer -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-kube-apiserver-operator/localhost-serving-signer" && oc -n openshift-kube-apiserver-operator get secret localhost-serving-signer -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-kube-apiserver-operator/service-network-serving-signer" && oc -n openshift-kube-apiserver-operator get secret service-network-serving-signer -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-kube-apiserver-operator/localhost-recovery-serving-signer" && oc -n openshift-kube-apiserver-operator get secret localhost-recovery-serving-signer -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-ingress-operator/router-ca" && oc -n openshift-ingress-operator get secret router-ca -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
~~~

~~~sh
openshift-kube-apiserver-operator/loadbalancer-serving-signer
notBefore=Oct 3 07:44:52 2022 GMT
notAfter=Sep 30 07:44:53 2032 GMT

openshift-kube-apiserver-operator/localhost-serving-signer
notBefore=Oct 3 07:44:52 2022 GMT
notAfter=Sep 30 07:44:53 2032 GMT

openshift-kube-apiserver-operator/service-network-serving-signer
notBefore=Oct 3 07:44:53 2022 GMT
notAfter=Sep 30 07:44:54 2032 GMT

openshift-kube-apiserver-operator/localhost-recovery-serving-signer
notBefore=Oct 3 07:44:53 2022 GMT
notAfter=Sep 30 07:44:54 2032 GMT

openshift-ingress-operator/router-ca
notBefore=Oct 3 07:47:26 2022 GMT
notAfter=Oct 2 07:47:27 2024 GMT
~~~

### Client Certs

~~~sh
echo "openshift-kube-apiserver/localhost-recovery-serving-certkey" && oc -n openshift-kube-apiserver get secret localhost-recovery-serving-certkey -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-kube-apiserver/internal-loadbalancer-serving-certkey" && oc -n openshift-kube-apiserver get secret internal-loadbalancer-serving-certkey -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-kube-apiserver/external-loadbalancer-serving-certkey" && oc -n openshift-kube-apiserver get secret external-loadbalancer-serving-certkey -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-kube-apiserver/localhost-serving-cert-certkey" && oc -n openshift-kube-apiserver get secret localhost-serving-cert-certkey -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-kube-apiserver/service-network-serving-certkey" && oc -n openshift-kube-apiserver get secret service-network-serving-certkey -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-ingress/router-certs-default" && oc -n openshift-ingress get secret router-certs-default -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
~~~

~~~sh
openshift-kube-apiserver/localhost-recovery-serving-certkey
notBefore=Oct 3 08:12:28 2022 GMT
notAfter=Sep 30 07:44:54 2032 GMT

openshift-kube-apiserver/internal-loadbalancer-serving-certkey
notBefore=Oct 3 08:12:28 2022 GMT
notAfter=Nov 2 08:12:29 2022 GMT

openshift-kube-apiserver/external-loadbalancer-serving-certkey
notBefore=Oct 3 08:12:28 2022 GMT
notAfter=Nov 2 08:12:29 2022 GMT

openshift-kube-apiserver/localhost-serving-cert-certkey
notBefore=Oct 3 08:12:28 2022 GMT
notAfter=Nov 2 08:12:29 2022 GMT

openshift-kube-apiserver/service-network-serving-certkey
notBefore=Oct 3 08:12:29 2022 GMT
notAfter=Nov 2 08:12:30 2022 GMT

openshift-ingress/router-certs-default
notBefore=Oct 3 09:50:35 2022 GMT
notAfter=Oct 2 09:50:36 2024 GMT
~~~

### CA Rotate

~~~sh
oc -n openshift-kube-apiserver-operator delete secret loadbalancer-serving-signer
oc -n openshift-kube-apiserver-operator delete secret localhost-serving-signer
oc -n openshift-kube-apiserver-operator delete secret service-network-serving-signer
oc -n openshift-kube-apiserver-operator delete secret localhost-recovery-serving-signer
oc -n openshift-ingress-operator delete secret router-ca
~~~

Cluster operators will roll out new revisions of the components. We need to wait until all cluster operators have stabilized.

> :warning: `openshift-ingress-operator/router-ca` secret may take a few moments to get created.

~~~sh
echo "openshift-kube-apiserver-operator/loadbalancer-serving-signer" && oc -n openshift-kube-apiserver-operator get secret loadbalancer-serving-signer -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-kube-apiserver-operator/localhost-serving-signer" && oc -n openshift-kube-apiserver-operator get secret localhost-serving-signer -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-kube-apiserver-operator/service-network-serving-signer" && oc -n openshift-kube-apiserver-operator get secret service-network-serving-signer -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-kube-apiserver-operator/localhost-recovery-serving-signer" && oc -n openshift-kube-apiserver-operator get secret localhost-recovery-serving-signer -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-ingress-operator/router-ca" && oc -n openshift-ingress-operator get secret router-ca -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
~~~

~~~sh
openshift-kube-apiserver-operator/loadbalancer-serving-signer
notBefore=Oct 4 17:39:59 2022 GMT
notAfter=Oct 1 17:40:00 2032 GMT

openshift-kube-apiserver-operator/localhost-serving-signer
notBefore=Oct 4 17:39:59 2022 GMT
notAfter=Oct 1 17:40:00 2032 GMT

openshift-kube-apiserver-operator/service-network-serving-signer
notBefore=Oct 4 17:39:59 2022 GMT
notAfter=Oct 1 17:40:00 2032 GMT

openshift-kube-apiserver-operator/localhost-recovery-serving-signer
notBefore=Oct 4 17:39:59 2022 GMT
notAfter=Oct 1 17:40:00 2032 GMT

openshift-ingress-operator/router-ca
notBefore=Oct 4 17:40:39 2022 GMT
notAfter=Oct 3 17:40:40 2024 GMT
~~~

At this point, we need to rotate the certificates issued by the old CA, otherwise the trust chain will be broken:

~~~sh
oc -n openshift-kube-apiserver-operator get secret loadbalancer-serving-signer -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/lb-signer.crt
oc -n openshift-kube-apiserver get secret internal-loadbalancer-serving-certkey -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/internallb.crt
~~~

~~~sh
openssl verify -verbose -CAfile /tmp/lb-signer.crt /tmp/internallb.crt
~~~

~~~sh
CN = api-int.mario-ipi.e2e.bos.redhat.com
error 20 at 0 depth lookup: unable to get local issuer certificate
error /tmp/internallb.crt: verification failed
~~~

As we explained earlier, even if the trust is broken with the new CA, the old CA is still present in the bundle.

Let's force the rotation of the issued certs:

> **NOTE**: We need to use `--insecure-skip-tls-verify=true` because once delete the lb certs the cert in the kubeconfig won't be valid anymore.

~~~sh
oc -n openshift-kube-apiserver delete secret localhost-recovery-serving-certkey --insecure-skip-tls-verify=true
oc -n openshift-kube-apiserver delete secret internal-loadbalancer-serving-certkey --insecure-skip-tls-verify=true
oc -n openshift-kube-apiserver delete secret external-loadbalancer-serving-certkey --insecure-skip-tls-verify=true
oc -n openshift-kube-apiserver delete secret localhost-serving-cert-certkey --insecure-skip-tls-verify=true
oc -n openshift-kube-apiserver delete secret service-network-serving-certkey --insecure-skip-tls-verify=true
oc -n openshift-ingress delete secret router-certs-default --insecure-skip-tls-verify=true
~~~

We need to wait for the cluster operators to rollout new revisions. At some point, cluster operators will degraded, it's possible that nodes degrade as well. We need to fix the cluster manually.

Our nodes will stop working since the CA Authority certificates in the kubeconfigs used by Kubelet on the nodes won't be valid anymore.

~~~sh
oc get nodes --insecure-skip-tls-verify=true
~~~

~~~sh
NAME STATUS ROLES AGE VERSION
openshift-master-0 NotReady master,worker 12d v1.24.0+b62823b
openshift-master-1 NotReady master,worker 12d v1.24.0+b62823b
openshift-master-2 NotReady master,worker 12d v1.24.0+b62823b
~~~

First, we fix the local kubeconfig (in our machine):

~~~sh
NEW_CRT=$(oc --insecure-skip-tls-verify=true -n openshift-kube-apiserver get secret external-loadbalancer-serving-certkey -o jsonpath='{.data.tls\.crt}')

sed -i.orig "s/certificate-authority-data: .*/certificate-authority-data: ${NEW_CRT}/g" /path/to/your/kubeconfig
~~~

Next, we fix the nodes:

~~~sh
NEW_CRT=$(oc --insecure-skip-tls-verify=true -n openshift-kube-apiserver get secret internal-loadbalancer-serving-certkey -o jsonpath='{.data.tls\.crt}')

ssh core@openshift-master-0 -- sudo sed -i.orig \"s/certificate-authority-data: .*/certificate-authority-data: ${NEW_CRT}/g\" /etc/kubernetes/kubeconfig
ssh core@openshift-master-0 -- sudo sed -i.orig \"s/certificate-authority-data: .*/certificate-authority-data: ${NEW_CRT}/g\" /var/lib/kubelet/kubeconfig
ssh core@openshift-master-1 -- sudo sed -i.orig \"s/certificate-authority-data: .*/certificate-authority-data: ${NEW_CRT}/g\" /etc/kubernetes/kubeconfig
ssh core@openshift-master-1 -- sudo sed -i.orig \"s/certificate-authority-data: .*/certificate-authority-data: ${NEW_CRT}/g\" /var/lib/kubelet/kubeconfig
ssh core@openshift-master-2 -- sudo sed -i.orig \"s/certificate-authority-data: .*/certificate-authority-data: ${NEW_CRT}/g\" /etc/kubernetes/kubeconfig
ssh core@openshift-master-2 -- sudo sed -i.orig \"s/certificate-authority-data: .*/certificate-authority-data: ${NEW_CRT}/g\" /var/lib/kubelet/kubeconfig

ssh core@openshift-master-0 -- sudo reboot
ssh core@openshift-master-1 -- sudo reboot
ssh core@openshift-master-2 -- sudo reboot
~~~

We need to wait for the cluster operators to move to a stable state.

You may see new CSRs being generated, if you do, approve them:

~~~sh
oc get csr
~~~

~~~
NAME AGE SIGNERNAME REQUESTOR REQUESTEDDURATION CONDITION
csr-4h5tj 75s kubernetes.io/kube-apiserver-client-kubelet system:node:openshift-master-2 <none> Pending
csr-dtptv 80s kubernetes.io/kube-apiserver-client-kubelet system:node:openshift-master-1 <none> Pending
csr-h5d7s 4m34s kubernetes.io/kube-apiserver-client-kubelet system:node:openshift-master-0 <none> Pending
~~~

~~~sh
oc get csr | grep Pending | awk '{print $1}' | xargs oc adm certificate approve
~~~

~~~sh
certificatesigningrequest.certificates.k8s.io/csr-4h5tj approved
certificatesigningrequest.certificates.k8s.io/csr-dtptv approved
certificatesigningrequest.certificates.k8s.io/csr-h5d7s approved
~~~

Once approved, nodes will get back to `Ready` state:

~~~sh
oc get nodes
~~~

~~~sh
NAME STATUS ROLES AGE VERSION
openshift-master-0 Ready master,worker 12d v1.24.0+b62823b
openshift-master-1 Ready master,worker 12d v1.24.0+b62823b
openshift-master-2 Ready master,worker 12d v1.24.0+b62823b
~~~

Now we need to wait for the cluster to stabilize, wait for all cluster operators to be ready again.

We can check the new certs expiration dates.

~~~sh
oc -n openshift-kube-apiserver get secret internal-loadbalancer-serving-certkey -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
~~~

~~~sh
notBefore=Oct 4 17:43:08 2022 GMT
notAfter=Nov 3 17:43:09 2022 GMT
~~~

~~~sh
oc -n openshift-kube-apiserver get secret internal-loadbalancer-serving-certkey -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/internallb.crt
~~~

~~~sh
openssl verify -verbose -CAfile /tmp/lb-signer.crt /tmp/internallb.crt
~~~

~~~sh
/tmp/internallb.crt: OK
~~~

We can check new validity periods:

~~~sh
echo "openshift-kube-apiserver/localhost-recovery-serving-certkey" && oc -n openshift-kube-apiserver get secret localhost-recovery-serving-certkey -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-kube-apiserver/internal-loadbalancer-serving-certkey" && oc -n openshift-kube-apiserver get secret internal-loadbalancer-serving-certkey -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-kube-apiserver/external-loadbalancer-serving-certkey" && oc -n openshift-kube-apiserver get secret external-loadbalancer-serving-certkey -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-kube-apiserver/localhost-serving-cert-certkey" && oc -n openshift-kube-apiserver get secret localhost-serving-cert-certkey -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-kube-apiserver/service-network-serving-certkey" && oc -n openshift-kube-apiserver get secret service-network-serving-certkey -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
echo "openshift-ingress/router-certs-default" && oc -n openshift-ingress get secret router-certs-default -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -startdate -enddate
~~~

~~~sh
openshift-kube-apiserver/localhost-recovery-serving-certkey
notBefore=Oct 3 08:12:28 2022 GMT
notAfter=Sep 30 07:44:54 2032 GMT

openshift-kube-apiserver/internal-loadbalancer-serving-certkey
notBefore=Oct 3 08:12:28 2022 GMT
notAfter=Nov 2 08:12:29 2022 GMT

openshift-kube-apiserver/external-loadbalancer-serving-certkey
notBefore=Oct 3 08:12:28 2022 GMT
notAfter=Nov 2 08:12:29 2022 GMT

openshift-kube-apiserver/localhost-serving-cert-certkey
notBefore=Oct 3 08:12:28 2022 GMT
notAfter=Nov 2 08:12:29 2022 GMT

openshift-kube-apiserver/service-network-serving-certkey
notBefore=Oct 3 08:12:29 2022 GMT
notAfter=Nov 2 08:12:30 2022 GMT

openshift-ingress/router-certs-default
notBefore=Oct 3 09:50:35 2022 GMT
notAfter=Oct 2 09:50:36 2024 GMT
~~~