Migrating to cilium v1.11.8 for OpenShift in Kube Proxy Replacement Mode.

Summary of changes.

With this release we are deploying a new configmap into the cilium namespace. The name of this configmap is cilium-ee-olm-overrides.

The purpose is two fold.

the default configmap looks like this:

apiVersion: v1
data:
  RELATED_IMAGE_CERTGEN: quay.io/cilium/certgen:v0.1.5@sha256:0c2b71bb3469990e7990e7e26243617aa344b5a69a4ce465740b8577f9d48ab9@sha256:0c2b71bb3469990e7990e7e26243617aa344b5a69a4ce465740b8577f9d48ab9
  RELATED_IMAGE_CILIUM: quay.io/isovalent/cilium-dev:v1.11.8-cee.1-iptables-stdout@sha256:c51cfc3d971a58172da62394c1eb08ce9d68c56a2ef237d9b0ab331791ee84e1
  RELATED_IMAGE_CILIUM_OPERATOR: quay.io/isovalent/operator-generic:v1.11.8-cee.1@sha256:948628cbf397ee14eda014e8a0d7ca06db9957f7ac47e6623a0480fe79b0ac40
  RELATED_IMAGE_CLUSTERMESH: quay.io/isovalent/clustermesh-apiserver:v1.11.8-cee.1@sha256:856339f0735c08f8ab7cc229e8d52e4b881a020128777b17527f0e62de990ed1
  RELATED_IMAGE_CLUSTERMESH_ETCD: quay.io/coreos/etcd:v3.4.13@sha256:04833b601fa130512450afa45c4fe484fee1293634f34c7ddc231bd193c74017@sha256:04833b601fa130512450afa45c4fe484fee1293634f34c7ddc231bd193c74017
  RELATED_IMAGE_ETCD_OPERATOR: quay.io/cilium/cilium-etcd-operator:v2.0.7@sha256:04b8327f7f992693c2cb483b999041ed8f92efc8e14f2a5f3ab95574a65ea2dc@sha256:04b8327f7f992693c2cb483b999041ed8f92efc8e14f2a5f3ab95574a65ea2dc
  RELATED_IMAGE_HUBBLE_ENTERPRISE: quay.io/isovalent/hubble-enterprise:v1.8.2@sha256:1e1d3e6c199546b2398f4166ff64210c04075bb407c8f07c5be8d3db08560d4a
  RELATED_IMAGE_HUBBLE_ENTERPRISE_METADATA: quay.io/isovalent/hubble-enterprise-metadata:current@sha256:372511f6fa3523dadfe9128caf77520cf74ce24df4a19e1583ee197375e06014
  RELATED_IMAGE_HUBBLE_ENTERPRISE_OPERATOR: quay.io/isovalent/hubble-enterprise-operator:v1.8.2@sha256:0d5e6a5d9c07780c18e5aa81d8f1c95c399bff2d36a44fa5f61ac89a788ca83b
  RELATED_IMAGE_HUBBLE_EXPORT: quay.io/cilium/hubble-export-stdout:v1.0.2@sha256:f1b1475a3c89950481264f5f8ca63eea3b911a45dc83fdf15122d695a0fba9b3
  RELATED_IMAGE_HUBBLE_EXPORT_FLUENTD: quay.io/isovalent/hubble-export-fluentd:v1.5.1@sha256:eaf42a84821509b69e2ef2b658796062e79a90e63045a98689da3da99a12e492
  RELATED_IMAGE_HUBBLE_EXPORT_S3: quay.io/isovalent/hubble-export-s3:2021-04-20-6f4b384@sha256:ce4f71f7f13301133b7d471b87fb81108b2c2da7fc29865ef96bc8431b2be72b
  RELATED_IMAGE_HUBBLE_RBAC: quay.io/isovalent/hubble-rbac:v1.1.0@sha256:9ae9187c25b1d49e9338c6ba4c73be18adee551da2bb9a342602d5060103ae11
  RELATED_IMAGE_HUBBLE_RELAY: quay.io/isovalent/hubble-relay:v1.11.8-cee.1@sha256:f4d9ce3a078dbebf93c9e5fb7d594d72cf9e7c4b52b4cfc912d87cfdb76a1e0b
  RELATED_IMAGE_HUBBLE_UI_BE: quay.io/isovalent/hubble-ui-enterprise-backend:v0.16.2@sha256:20364df292b5047f14c48805e6a62918bd32d7f2a36f34480cafc8c9ea3a5c9a
  RELATED_IMAGE_HUBBLE_UI_FE: quay.io/isovalent/hubble-ui-enterprise:v0.16.2@sha256:7af8f63d1b3844d65606374fbd770e09b4548915bb3d556a358d2b21aeb4bed1
  RELATED_IMAGE_HUBBLE_UI_OAUTH: quay.io/oauth2-proxy/oauth2-proxy:v7.1.3@sha256:ecd26b74a01f2b547ddaed4d32d35f8f5e09c378d5c1fc6cfa63f0adf659ac2b
  RELATED_IMAGE_NODEINIT: quay.io/cilium/startup-script:62bfbe88c17778aad7bef9fa57ff9e2d4a9ba0d8@sha256:1daf817f34000399fcb5da9a94cb299e2810d2c7a52e51de22ba0d4783b6ce84
  RELATED_IMAGE_PREFLIGHT: quay.io/isovalent/cilium-dev:v1.11.8-cee.1-iptables-stdout@sha256:c51cfc3d971a58172da62394c1eb08ce9d68c56a2ef237d9b0ab331791ee84e1
kind: ConfigMap
metadata:
  labels:
    name: cilium-ee-olm
  name: cilium-ee-olm-overrides
  namespace: cilium

This allows us to override the cilium images used by the cilium-ee-olm operator by specifying them differently in the configmap.

You can move to this model if you want and remove the references to the repository mapping in the ciliumconfig used currently.

Important Note:

The environment variables defined in the cilium-ee-olm-overrides configmap will take precedence over the things defined in the ciliumconfig.

If you want to keep setting the repository value with the ciliumconfig resource you should remove the variables above from the configmap.

When deploying in kube-proxy-replacement mode strict we also need to configure the cilium-ee-olm operator and override the KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT environment variables to point directly at a load balancer or fqdn in front of the openshift cluster.

We can use this cilium-ee-olm-overrides configmap to store this data as well.

If you want to continue to use the ciliumconfig to manage the repository mapping your resultant cilium-ee-olm-overides configmap would look like this:

apiVersion: v1
data:
  KUBERNETES_SERVICE_HOST: "api-int.ocp1.k8s.work"
  KUBERNETES_SERVICE_PORT: "6443"
kind: ConfigMap
metadata:
  labels:
    name: cilium-ee-olm
  name: cilium-ee-olm-overrides
  namespace: cilium

This would allow the repository value in the ciliumconfig to take precendence and also set the KUBERNETES_SERVICE env vars correctly.

Note: that you would want to use the SERVICE_HOST for your target cluster.

The cilium-ee-olm image can be pulled from the dock

Images:

The cilium-ee-olm image can be pulled from here:

quay.io/isovalent-dev/cilium-ee-olm:8ac096d0b7584121370e847c98a6ac278ac95299-v1.11.8-image-override-fix

and then updated in the deployment manifest to point toward a private repository.

quay.io/cilium/certgen:v0.1.5
quay.io/isovalent/cilium-dev:v1.11.8-cee.1-iptables-stdout
quay.io/isovalent/operator-generic:v1.11.8-cee.1
quay.io/isovalent/clustermesh-apiserver:v1.11.8-cee.1
quay.io/coreos/etcd:v3.4.13
quay.io/cilium/cilium-etcd-operator:v2.0.7
quay.io/isovalent/hubble-enterprise:v1.8.2
quay.io/isovalent/hubble-enterprise-metadata:current
quay.io/isovalent/hubble-enterprise-operator:v1.8.2
quay.io/cilium/hubble-export-stdout:v1.0.2
quay.io/isovalent/hubble-export-fluentd:v1.5.1
quay.io/isovalent/hubble-export-s3:2021-04-20-6f4b384
quay.io/isovalent/hubble-rbac:v1.1.0
quay.io/isovalent/hubble-relay:v1.11.8-cee.1
quay.io/isovalent/hubble-ui-enterprise-backend:v0.16.2
quay.io/isovalent/hubble-ui-enterprise:v0.16.2
quay.io/oauth2-proxy/oauth2-proxy:v7.1.3
quay.io/cilium/startup-script:62bfbe88c17778aad7bef9fa57ff9e2d4a9ba0d8
quay.io/isovalent/cilium-dev:v1.11.8-cee.1-iptables-stdout

Manifests.

cilium.v1.11.8_1.zip

Upgrade Procedure:

Download the manifests locally.

Apply the necessary changes to the cilium.v1.11.8_1/cluster-network-06-cilium-00002-cilium-ee-olm-overrides-configmap.yaml
configmap.

Apply the manifests to the cluster with kubectl apply -f cilium.v1.11.8_1/

After upgrade you should see the following:

cilium status -n cilium

    /¯¯\
 /¯¯\__/¯¯\    Cilium:         OK
 \__/¯¯\__/    Operator:       OK
 /¯¯\__/¯¯\    Hubble:         OK
 \__/¯¯\__/    ClusterMesh:    disabled
    \__/

DaemonSet         cilium             Desired: 4, Ready: 4/4, Available: 4/4
Deployment        cilium-operator    Desired: 2, Ready: 2/2, Available: 2/2
Deployment        hubble-relay       Desired: 1, Ready: 1/1, Available: 1/1
Deployment        hubble-ui          Desired: 1, Ready: 1/1, Available: 1/1
Containers:       hubble-ui          Running: 1
                  cilium             Running: 4
                  cilium-operator    Running: 2
                  hubble-relay       Running: 1
Cluster Pods:     79/123 managed by Cilium
Image versions    cilium             quay.io/isovalent/cilium-dev:v1.11.8-cee.1-iptables-stdout: 4
                  cilium-operator    quay.io/isovalent/operator-generic:v1.11.8-cee.1: 2
                  hubble-relay       quay.io/isovalent/hubble-relay:v1.11.8-cee.1: 1
                  hubble-ui          quay.io/isovalent/hubble-ui-enterprise-backend:v0.16.2: 1
                  hubble-ui          quay.io/isovalent/hubble-ui-enterprise:v0.16.2: 1