# k8s-upgrade-1.22 # dr - what is not working? ### aws-node: ```Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 10.32.39.155 10.32.97.63 10.32.161.253``` ``` Failed to load logs: Get "https://10.96.111.8:10250/containerLogs/kube-system/aws-node-d8gx2/aws-node?tailLines=502&timestamps=true": dial tcp 10.96.111.8:10250: connect: connection refused ``` ### kube-prorxy: ``` Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 10.32.39.155 10.32.97.63 10.32.161.253 ``` **upgrade metrics-server** **(STG)** **upgrade kube-proxy** **(DR, STG)** https://docs.aws.amazon.com/eks/latest/userguide/managing-kube-proxy.html **Enable Prometheus Monitoring on kube-proxy: (DR, STG)** Add via: ``` kubectl --namespace kube-system edit daemonsets.apps kube-proxy ``` Ports section: ``` ports: - containerPort: 10249 hostPort: 10249 name: metrics protocol: TCP ``` ``` kubectl --namespace kube-system edit configmaps kube-proxy-config ``` Change: ``` metricsBindAddress: 127.0.0.1:10249 ``` To: ``` metricsBindAddress: 0.0.0.0:10249 ``` ### coredns ``` Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 10.32.39.155 10.32.97.63 10.32.161.253 ``` **upgrade coredns** **(DR, STG)** current version: `v1.8.0-eksbuild.1` upgrade to: `v1.8.7-eksbuild.3` https://docs.aws.amazon.com/eks/latest/userguide/managing-coredns.html ### aws-node (AWS CNI) **upgrade aws-node** (DR, STG) current version: `v1.11.4` upgrade to: `1.12.1-eksbuild.1` https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html ### kubectl `error: exec plugin: invalid apiVersion "client.authentication.k8s.io/v1alpha1"` this needs to be fixed from `kube/config` kubectl client version `v1.25.5` does not work. according to google, we may need to use `v.1.24.x` kubectl version You need to update your AWS CLI to >2.7.25 or the latest (recommended), ensure your CLI is pointing to the right region, then try eksctl utils write-kubeconfig --cluster=NAME. Open the kubeconfig file and check client.authentication.k8s.io/v1alpha1 has changed to client.authentication.k8s.io/v1beta1. ### cluster-autoscaler **(DR, STG)** Needs to match k8s cluster version: `k8s version v1.22` => `autoscaler v1.22` ### efs-provisioner (DR) ``` efs-provisioner.go:91] couldn't confirm that the EFS file system exists: AccessDeniedException: User: arn:aws:sts::152638385147:assumed-role/nodes-drnode/i-0c05e2bff43c966b0 is not authorized to perform: elasticfilesystem:DescribeFileSystems on the specified resource status code: 403, request id: 7af38b70-dcf2-4bdd-8abc-e38c10243fd5 ``` Fixed with following changes: https://github.com/dicefm/terraform-module-eks/tree/eksup https://github.com/dicefm/terraformpds/pull/216 ### ingress-nginx-controller (SKIPPING) Check the version compatibility chart: https://github.com/kubernetes/ingress-nginx According to the above chart, we should install ingress-nginx-controller `v1.4.0`. Current installed version is `v1.5.1` which is not suggested for Kubernetes version `1.22` ### kube-prometheus-kube-state-metrics ``` E0123 15:02:30.138739 1 reflector.go:156] pkg/mod/k8s.io/client-go@v0.0.0-20191109102209-3c0d1af94be5/tools/cache/reflector.go:108: Failed to list *v1beta1.CertificateSigningRequest: the server could not find the requested resource E0123 15:02:51.179480 1 reflector.go:156] pkg/mod/k8s.io/client-go@v0.0.0-20191109102209-3c0d1af94be5/tools/cache/reflector.go:108: Failed to list *v1beta1.Ingress: the server could not find the requested resource (get ingresses.extensions) ``` ### MutatingWebhookConfiguration **infra-kubernetes-charts** spark-operator/backup/ote.yaml **(skip)** spark-operator/backup/prod.yaml **(skip)** spark-operator/backup/stg.yaml **(skip)** `admissionregistration.k8s.io/v1beta1` > `admissionregistration.k8s.io/v1` ### APIService **infra-kubernetes-charts** datadog/helm-charts/datadog6/charts_copy/datadog6/templates/agent-apiservice.yaml `apiregistration.k8s.io/v1beta1` > `apiregistration.k8s.io/v1` ### Ingress **infra-kubernetes-charts** keycloak/helm-charts/keycloak-access/templates/ingress.yaml **(THAT IS ON DC)** `networking.k8s.io/v1beta1` > `networking.k8s.io/v1` nginx-ingress/rules/blogdice.fm **(DR)** nginx-ingress/rules/branddice.fm **(DR)** nginx-ingress/rules/nitedice.yaml **(DR)** puppet/helm-charts/proxy/templates/ingress.yaml **(THAT IS ON DC)** vaultwarden/helm-charts/vaultwarden/templates/ingress.yaml **(THAT IS ON DC)** ### PriorityClass **infra-kubernetes-charts** `scheduling.k8s.io/v1beta1` > `scheduling.k8s.io/v1`