# Kubernetes Cert Renewal ## What Certs are affected? Kubernetes-ca.pem kubernetes-ca-key.pem kubernetes-api-cert.pem kubernetes-api-key.pem kube-scheduler.pem kube-scheduler-key.pem kube-controller-manager-key.pem kube-controller-manager.pem etcd-ca.pem etcd-client-cert.pem etcd-client-key.pem ## What happens when the Kubernetes certs expires? Communication between the Kubernetes control plane components and the Kubelets to communicate with each other. The certifiactes provide authentication to ensure that the communication coming to and from Kubernetes components are valid for the cluster. When a components certificate expires it can no longer communicate with the rest of the cluster. This means that any change that occurs or is expected to occur will no longer take place. What happens when each components cert expires: * API Server * Requests coming into the cluster will not function as expected. The API Server can no longer communicated with the ETCD server to retrieve or input data. * Controller Manager * The Controller Manager can no longer receive updates from the API Server, keeping the CM unaware of the current state of the cluster. * Scheduler * If the Scheduler would be able to get updates from the Controller Manager it still wouldn't be able to communicate those changes to the Kubelets * ETCD * The ETCD nodes would no longer be able to function as a cluster and would be unable to communicate with the API Server. An ETCD cluster would become seperated into three or more stand alone ETCD nodes. If the nodes are unable to communicate with each other over a long enough period of time they will not be able to join back together to form a cluster. * Kubelet * Nodes would be unable to communicate their current state back to the API Server, nor would they be able to receive updates from the Scheduler. All activity on the node would stay the same until the certs were renewed. As we can see the effect of certificates expiring is catasptrophic to a Kubernetes cluster. The silver lining to this rain cloud is that while all communication in a cluster is down when the certs expire the current running pods will continue to run. This only becomes an outtage to running pods if for any reason the node running the pods decides they need to be rescheduled or the pod crashes on accident. ## How often do certs expire? everything but cas 1 year cas 10 years NL: Is that the case these days? It used to be that CAs expired in 1 year as well? If that's changed then that's awesome ## How to fix expired certs ### Kubeadm damn i cant paste on my phone at a reataurant in barcelona with @hhoover :) YOOOOOOOO! Say what up to Hoover. Also, have some sangria for me kubeadm as of 1.15 will rotate client certs as part of upgrade kubeadm alpha certs check-expiration ### Manually omg why NL: We gotta, not everyone uses kubeadm (cough wayfair cough) its worth using it for this reason alone NL: You don't have to convince me, they've built the automation in place without it. I'm going to change that on the next upgrade cycle cause 1.16 be crazy anyway :). really tho: how did you create the certz? do that again :) mind the sans tho they are important!! ## How to avoid this in the future? use prom to measure and alert on when certs expire. NL: don't you mean wavefront :P rotate them before that!