Fresh install and upgrade paths tested for openshift.

Upgrade paths tested.

OpenShift 4.10.20 -> 4.11.20
cilium 1.11.8 -> 1.11.9

openshift 4.10.20 -> 4.11.20
cilium 1.11.8 -> 1.12.3

openshift 4.11.2
cilium 1.12.3 -> 1.12.6

https://hackmd.io/@mauilion/openshift_install

Instructions for fresh install of OpenShift.

When installing a fresh OpenStack cluster in Kube Proxy Replacement mode you would follow the standard installation steps laid out by the OpenShift documentation. When you have reached the openshift-install create manifests stage. You will need to make the following changes.

Download the manifests for the cilium-ee-olm version you want to install.

expand the content into the manifests/ directory.

Make the following changes to the content of the manifests directory.

Here is a video walk through of a fresh install procedure.

cluster-network-02-operator.yml

This file will determin if kube proxy is deployed as part of the OpenShift stack. You can copy the existing cluster-network-02-config.yml to cluster-network-02-operator.yml and make the following changes.

apiVersion: operator.openshift.io/v1 # change config.openshift.io/v1 to operator.openshift.io/v1
kind: Network
metadata:
  creationTimestamp: null
  name: cluster
spec:
  clusterNetwork:
  - cidr: 10.254.0.0/16
    hostPrefix: 24
  externalIP:
    policy: {}
  networkType: Cilium
  deployKubeProxy: false # add the new deployKubeProxy: false line. 
  serviceNetwork:
  - 172.30.0.0/16
status: {}

cluster-network-06-cilium-00002-cilium-ee-olm-overrides-configmap.yaml

This file contains overrides for the cilium ee operator. If you want to change this file to point to images hosted within Visa that would be great.

You can also add a couple of environment variables that point to the apiserver load balancer so that we can handle the install when running without kube proxy.

The config with the environment varialbes would look like this:

apiVersion: v1
data:
  KUBERNETES_SERVICE_HOST: "api-int.ocp1.k8s.work"
  KUBERNETES_SERVICE_PORT: "6443"
kind: ConfigMap
metadata:
  labels:
    name: cilium-ee-olm
  name: cilium-ee-olm-overrides
  namespace: cilium

cluster-network-07-cilium-ciliumconfig.yaml

For a fresh install you can use the cilium-ee-olm-overrides document above to specify where the images will be pulled from since this has already been handled this simplified the ciliumconfig considerably.

Here is an example ciliumconfig. This example isn't complete as it doesn't include the log forwarding configuration.

apiVersion: cilium.io/v1alpha1
kind: CiliumConfig
metadata:
  name: cilium-enterprise
  namespace: cilium
spec:
  cilium:
    cni:
      binPath: "/var/lib/cni/bin"
      confPath: "/var/run/multus/cni/net.d"
    kubeProxyReplacement: strict
    k8sServiceHost: api-int.ocp1.k8s.work
    k8sServicePort: 6443
    extraConfig:
      bpf-lb-sock-hostns-only: "true"
      export-aggregation: "connection"
      export-aggregation-ignore-source-port: "false"
      export-aggregation-state-filter: "new closed established error"
    prometheus:
      enabled: true
    metrics:
      enabled: true
    operator:
      metrics:
        enabled: true
    hubble:
      enabled: true
      metrics:
        enabled:
        - dns:query;ignoreAAAA
        - drop:sourceContext=identity;destinationContext=identity
        - tcp
        - flow
        - port-distribution
        - icmp
        - http
      relay:
        enabled: true
    nodeinit:
      enabled: true
    ipam:
      mode: cluster-pool
      operator:
        clusterPoolIPv4PodCIDR: "10.254.0.0/16"
        clusterPoolIPv4MaskSize: 24
  hubble-enterprise:
    metrics:
      enabled: true
    prometheus:
      enabled: true
    enabled: true
    enterprise:
      enabled: true
  hubble-ui:
    prometheus:
      enabled: true
    metrics:
      enabled: true
    enabled: true

Resources 1.11.9

Manifests: https://docs.isovalent.com/v1.12/public/cilium-ee-olm/cilium-ee-1.11.9.tar.gz

Checksum: https://docs.isovalent.com/v1.12/public/cilium-ee-olm/cilium-ee-1.11.9.tar.gz.sha256

Images:

quay.io/cilium/certgen:v0.1.5@sha256:0c2b71bb3469990e7990e7e26243617aa344b5a69a4ce465740b8577f9d48ab9@sha256
 quay.io/cilium/cilium-etcd-operator:v2.0.7@sha256:04b8327f7f992693c2cb483b999041ed8f92efc8e14f2a5f3ab95574a65ea2dc@sha256
 quay.io/cilium/hubble-export-stdout:v1.0.2@sha256:f1b1475a3c89950481264f5f8ca63eea3b911a45dc83fdf15122d695a0fba9b3
 quay.io/cilium/startup-script:62bfbe88c17778aad7bef9fa57ff9e2d4a9ba0d8@sha256:1daf817f34000399fcb5da9a94cb299e2810d2c7a52e51de22ba0d4783b6ce84
quay.io/coreos/etcd:v3.4.13@sha256:04833b601fa130512450afa45c4fe484fee1293634f34c7ddc231bd193c74017@sha256
 quay.io/isovalent/cilium:v1.11.9-cee.1@sha256:24303e0eb4d7023bb665b37c4b07adb75e2cd995c24d1b15849f17ed4498bce1
 quay.io/isovalent/clustermesh-apiserver:v1.11.9-cee.1@sha256:257529c1e73b3628e8701e712bb76455b81ca36b68943299d93f2a569218cb3e
 quay.io/isovalent/hubble-enterprise-metadata:current@sha256:372511f6fa3523dadfe9128caf77520cf74ce24df4a19e1583ee197375e06014
 quay.io/isovalent/hubble-enterprise-operator:v1.8.2@sha256:0d5e6a5d9c07780c18e5aa81d8f1c95c399bff2d36a44fa5f61ac89a788ca83b
 quay.io/isovalent/hubble-enterprise:v1.8.2@sha256:1e1d3e6c199546b2398f4166ff64210c04075bb407c8f07c5be8d3db08560d4a
 quay.io/isovalent/hubble-export-fluentd:v1.5.1@sha256:eaf42a84821509b69e2ef2b658796062e79a90e63045a98689da3da99a12e492
 quay.io/isovalent/hubble-export-s3:2021-04-20-6f4b384@sha256:ce4f71f7f13301133b7d471b87fb81108b2c2da7fc29865ef96bc8431b2be72b
 quay.io/isovalent/hubble-rbac:v1.1.0@sha256:9ae9187c25b1d49e9338c6ba4c73be18adee551da2bb9a342602d5060103ae11
 quay.io/isovalent/hubble-relay:v1.11.9-cee.1@sha256:99498c03e9b48ae6ba72f7a811acd7b0c160a014d503be35007ee97b70260e69
 quay.io/isovalent/hubble-ui-enterprise-backend:v0.17.0@sha256:ecc0212aa5442744c147a0f49817ac4d1954eaf8e19edf538a7b4e3ef2c24423
 quay.io/isovalent/hubble-ui-enterprise:v0.17.0@sha256:e75817a960b04ae012c3e9c0c664fdab51840109524fb721c3b35e3392f14aa5
 quay.io/isovalent/operator-generic:v1.11.9-cee.1@sha256:1aaabd1bd3e64053a525c5f89fd7608faa0f6504fd9171944259561445c98d67
 quay.io/oauth2-proxy/oauth2-proxy:v7.1.3@sha256:ecd26b74a01f2b547ddaed4d32d35f8f5e09c378d5c1fc6cfa63f0adf659ac2b

Resources 1.12.3

Manifests: https://docs.isovalent.com/v1.12/public/cilium-ee-olm/cilium-ee-1.12.3.tar.gz

Checksum: https://docs.isovalent.com/v1.12/public/cilium-ee-olm/cilium-ee-1.12.3.tar.gz.sha256

Images:

quay.io/cilium/certgen:v0.1.8@sha256:4a456552a5f192992a6edcec2febb1c54870d665173a33dc7d876129b199ddbd
 quay.io/cilium/cilium-etcd-operator:v2.0.7@sha256:04b8327f7f992693c2cb483b999041ed8f92efc8e14f2a5f3ab95574a65ea2dc
 quay.io/cilium/hubble-export-stdout:v1.0.2@sha256:f1b1475a3c89950481264f5f8ca63eea3b911a45dc83fdf15122d695a0fba9b3
 quay.io/cilium/startup-script:d69851597ea019af980891a4628fb36b7880ec26@sha256:c263d1678fb426842c0836358c1da7628d771126211694a3776c4b8500cbb215
 quay.io/coreos/etcd:v3.5.4@sha256:795d8660c48c439a7c3764c2330ed9222ab5db5bb524d8d0607cac76f7ba82a3
 quay.io/isovalent/cilium:v1.12.3-cee.1@sha256:719d2fa581d8bc8b32eae4f737799f0d42b106eb213edcb53ed0dd88188a594c
 quay.io/isovalent/clustermesh-apiserver:v1.12.3-cee.1@sha256:765c3d0b4becf17dbbb172ba2f4445b3a52ea2c407c46e032c4cb1d8a7a4e060
 quay.io/isovalent/hubble-enterprise-metadata:current@sha256:372511f6fa3523dadfe9128caf77520cf74ce24df4a19e1583ee197375e06014
 quay.io/isovalent/hubble-enterprise-operator:v1.8.2@sha256:0d5e6a5d9c07780c18e5aa81d8f1c95c399bff2d36a44fa5f61ac89a788ca83b
 quay.io/isovalent/hubble-enterprise:v1.8.2@sha256:1e1d3e6c199546b2398f4166ff64210c04075bb407c8f07c5be8d3db08560d4a
 quay.io/isovalent/hubble-export-fluentd:v1.5.1@sha256:eaf42a84821509b69e2ef2b658796062e79a90e63045a98689da3da99a12e492
 quay.io/isovalent/hubble-export-s3:2021-04-20-6f4b384@sha256:ce4f71f7f13301133b7d471b87fb81108b2c2da7fc29865ef96bc8431b2be72b
 quay.io/isovalent/hubble-rbac:v1.1.0@sha256:9ae9187c25b1d49e9338c6ba4c73be18adee551da2bb9a342602d5060103ae11
 quay.io/isovalent/hubble-relay:v1.12.3-cee.1@sha256:03f67eb9e487d007c6992193eb462846bf9476d91dc710d146ff93bd71de0c97
 quay.io/isovalent/hubble-ui-enterprise-backend:v0.17.2@sha256:4663af651286050923571bf0ffcd8eb6bfe852af9dc2b9b4c5050a7a23b131e8
 quay.io/isovalent/hubble-ui-enterprise:v0.17.2@sha256:498fc2aa10d45f19b18b42ea7814567965b739cfb60a62c4ad6671d3f25db570
 quay.io/isovalent/operator-generic:v1.12.3-cee.1@sha256:3262b407f7aa70bf397b4c398848e6204f923ade1ed0512e66461e2e14d76347
 quay.io/oauth2-proxy/oauth2-proxy:v7.1.3@sha256:ecd26b74a01f2b547ddaed4d32d35f8f5e09c378d5c1fc6cfa63f0adf659ac2b

Resources 1.12.6

Manifests: https://docs.isovalent.com/v1.12/public/cilium-ee-olm/cilium-ee-1.12.6.tar.gz

Checksum: https://docs.isovalent.com/v1.12/public/cilium-ee-olm/cilium-ee-1.12.6.tar.gz.sha256

Images:

quay.io/cilium/certgen:v0.1.8@sha256:4a456552a5f192992a6edcec2febb1c54870d665173a33dc7d876129b199ddbd@sha256:2e0fc99ace29bbe5837718684920112a026bffc46f70efb38ea596c524e68819
quay.io/cilium/cilium-etcd-operator:v2.0.7@sha256:04b8327f7f992693c2cb483b999041ed8f92efc8e14f2a5f3ab95574a65ea2dc@sha256:04b8327f7f992693c2cb483b999041ed8f92efc8e14f2a5f3ab95574a65ea2dc
quay.io/cilium/startup-script:d69851597ea019af980891a4628fb36b7880ec26@sha256:c263d1678fb426842c0836358c1da7628d771126211694a3776c4b8500cbb215
quay.io/coreos/etcd:v3.5.4@sha256:795d8660c48c439a7c3764c2330ed9222ab5db5bb524d8d0607cac76f7ba82a3@sha256:a67fb152d4c53223e96e818420c37f11d05c2d92cf62c05ca5604066c37295e9
quay.io/isovalent/cilium:v1.12.6-cee.1@sha256:be14c06a64d3735c547591a0da042d6d63ccde843204b2bfa491b877ac0ac6ab
quay.io/isovalent/clustermesh-apiserver:v1.12.6-cee.1@sha256:80045bf094cef749277dcd7a18e7e082d9a049798d9fea3e95e2c40b7f501931
quay.io/isovalent/hubble-enterprise-metadata:current@sha256:372511f6fa3523dadfe9128caf77520cf74ce24df4a19e1583ee197375e06014
quay.io/isovalent/hubble-enterprise-operator:v1.8.5@sha256:216898c8516c436347b94ef528c5a8bbd0f86c0d66b86688a0b56be3f83048f1
quay.io/isovalent/hubble-enterprise:v1.8.5@sha256:e4b05dade11d665f6ae9f6cbb8f203ae185e967d1df0c1fdddb5604a95577ad5
quay.io/isovalent/hubble-export-fluentd:v1.5.1@sha256:eaf42a84821509b69e2ef2b658796062e79a90e63045a98689da3da99a12e492
quay.io/isovalent/hubble-export-s3:2021-04-20-6f4b384@sha256:ce4f71f7f13301133b7d471b87fb81108b2c2da7fc29865ef96bc8431b2be72b
quay.io/isovalent/hubble-export-stdout:v1.0.2@sha256:ca42970d2909e57762ac1c2386d1fa74171c181eb83922f0d53ae44f016b3e6d
quay.io/isovalent/hubble-rbac:v1.2.0@sha256:0d6da624a03183ea6905aba5bcf12dd9765c60fbc34d861d567de49827b1d2cd
quay.io/isovalent/hubble-relay:v1.12.6-cee.1@sha256:26c4e21d908b1aba1e562cffdfd1af7baf88f6f0d6186b1e79aa4f5fc3588916
quay.io/isovalent/hubble-ui-enterprise-backend:v0.17.6@sha256:ede27aecbd2c4d622d1450c91a48ebf01f5606d4751ac4230c409382821d47cf
quay.io/isovalent/hubble-ui-enterprise:v0.17.6@sha256:e75323d008bf6bf18c4419266edfa4dc2b98d646924827d9d382fcf59fb682df
quay.io/isovalent/operator-generic:v1.12.6-cee.1@sha256:1dbad6ca9134ac06f5f2882a13957b32c9510737c2c8c521234c38a6bbc7cd4e
quay.io/oauth2-proxy/oauth2-proxy:v7.1.3@sha256:ecd26b74a01f2b547ddaed4d32d35f8f5e09c378d5c1fc6cfa63f0adf659ac2b