# Fresh install and upgrade paths tested for openshift. ## Upgrade paths tested. OpenShift 4.10.20 -> 4.11.20 cilium 1.11.8 -> 1.11.9 openshift 4.10.20 -> 4.11.20 cilium 1.11.8 -> 1.12.3 openshift 4.11.2 cilium 1.12.3 -> 1.12.6 https://hackmd.io/@mauilion/openshift_install ## Instructions for fresh install of OpenShift. When installing a fresh OpenStack cluster in Kube Proxy Replacement mode you would follow the standard installation steps laid out by the OpenShift documentation. When you have reached the `openshift-install create manifests` stage. You will need to make the following changes. Download the manifests for the cilium-ee-olm version you want to install. expand the content into the `manifests/` directory. Make the following changes to the content of the manifests directory. Here is a video walk through of a fresh install procedure. {%youtube kAN2xgJS9aA %} #### cluster-network-02-operator.yml This file will determin if kube proxy is deployed as part of the OpenShift stack. You can copy the existing `cluster-network-02-config.yml` to `cluster-network-02-operator.yml` and make the following changes. ``` yaml apiVersion: operator.openshift.io/v1 # change config.openshift.io/v1 to operator.openshift.io/v1 kind: Network metadata: creationTimestamp: null name: cluster spec: clusterNetwork: - cidr: 10.254.0.0/16 hostPrefix: 24 externalIP: policy: {} networkType: Cilium deployKubeProxy: false # add the new deployKubeProxy: false line. serviceNetwork: - 172.30.0.0/16 status: {} ``` #### cluster-network-06-cilium-00002-cilium-ee-olm-overrides-configmap.yaml This file contains overrides for the cilium ee operator. If you want to change this file to point to images hosted within Visa that would be great. You can also add a couple of environment variables that point to the apiserver load balancer so that we can handle the install when running without kube proxy. The config with the environment varialbes would look like this: ``` yaml apiVersion: v1 data: KUBERNETES_SERVICE_HOST: "api-int.ocp1.k8s.work" KUBERNETES_SERVICE_PORT: "6443" kind: ConfigMap metadata: labels: name: cilium-ee-olm name: cilium-ee-olm-overrides namespace: cilium ``` #### cluster-network-07-cilium-ciliumconfig.yaml For a fresh install you can use the cilium-ee-olm-overrides document above to specify where the images will be pulled from since this has already been handled this simplified the ciliumconfig considerably. Here is an example ciliumconfig. This example isn't complete as it doesn't include the log forwarding configuration. ``` yaml apiVersion: cilium.io/v1alpha1 kind: CiliumConfig metadata: name: cilium-enterprise namespace: cilium spec: cilium: cni: binPath: "/var/lib/cni/bin" confPath: "/var/run/multus/cni/net.d" kubeProxyReplacement: strict k8sServiceHost: api-int.ocp1.k8s.work k8sServicePort: 6443 extraConfig: bpf-lb-sock-hostns-only: "true" export-aggregation: "connection" export-aggregation-ignore-source-port: "false" export-aggregation-state-filter: "new closed established error" prometheus: enabled: true metrics: enabled: true operator: metrics: enabled: true hubble: enabled: true metrics: enabled: - dns:query;ignoreAAAA - drop:sourceContext=identity;destinationContext=identity - tcp - flow - port-distribution - icmp - http relay: enabled: true nodeinit: enabled: true ipam: mode: cluster-pool operator: clusterPoolIPv4PodCIDR: "10.254.0.0/16" clusterPoolIPv4MaskSize: 24 hubble-enterprise: metrics: enabled: true prometheus: enabled: true enabled: true enterprise: enabled: true hubble-ui: prometheus: enabled: true metrics: enabled: true enabled: true ``` ### Resources 1.11.9 Manifests: https://docs.isovalent.com/v1.12/public/cilium-ee-olm/cilium-ee-1.11.9.tar.gz Checksum: https://docs.isovalent.com/v1.12/public/cilium-ee-olm/cilium-ee-1.11.9.tar.gz.sha256 Images: ``` quay.io/cilium/certgen:v0.1.5@sha256:0c2b71bb3469990e7990e7e26243617aa344b5a69a4ce465740b8577f9d48ab9@sha256 quay.io/cilium/cilium-etcd-operator:v2.0.7@sha256:04b8327f7f992693c2cb483b999041ed8f92efc8e14f2a5f3ab95574a65ea2dc@sha256 quay.io/cilium/hubble-export-stdout:v1.0.2@sha256:f1b1475a3c89950481264f5f8ca63eea3b911a45dc83fdf15122d695a0fba9b3 quay.io/cilium/startup-script:62bfbe88c17778aad7bef9fa57ff9e2d4a9ba0d8@sha256:1daf817f34000399fcb5da9a94cb299e2810d2c7a52e51de22ba0d4783b6ce84 quay.io/coreos/etcd:v3.4.13@sha256:04833b601fa130512450afa45c4fe484fee1293634f34c7ddc231bd193c74017@sha256 quay.io/isovalent/cilium:v1.11.9-cee.1@sha256:24303e0eb4d7023bb665b37c4b07adb75e2cd995c24d1b15849f17ed4498bce1 quay.io/isovalent/clustermesh-apiserver:v1.11.9-cee.1@sha256:257529c1e73b3628e8701e712bb76455b81ca36b68943299d93f2a569218cb3e quay.io/isovalent/hubble-enterprise-metadata:current@sha256:372511f6fa3523dadfe9128caf77520cf74ce24df4a19e1583ee197375e06014 quay.io/isovalent/hubble-enterprise-operator:v1.8.2@sha256:0d5e6a5d9c07780c18e5aa81d8f1c95c399bff2d36a44fa5f61ac89a788ca83b quay.io/isovalent/hubble-enterprise:v1.8.2@sha256:1e1d3e6c199546b2398f4166ff64210c04075bb407c8f07c5be8d3db08560d4a quay.io/isovalent/hubble-export-fluentd:v1.5.1@sha256:eaf42a84821509b69e2ef2b658796062e79a90e63045a98689da3da99a12e492 quay.io/isovalent/hubble-export-s3:2021-04-20-6f4b384@sha256:ce4f71f7f13301133b7d471b87fb81108b2c2da7fc29865ef96bc8431b2be72b quay.io/isovalent/hubble-rbac:v1.1.0@sha256:9ae9187c25b1d49e9338c6ba4c73be18adee551da2bb9a342602d5060103ae11 quay.io/isovalent/hubble-relay:v1.11.9-cee.1@sha256:99498c03e9b48ae6ba72f7a811acd7b0c160a014d503be35007ee97b70260e69 quay.io/isovalent/hubble-ui-enterprise-backend:v0.17.0@sha256:ecc0212aa5442744c147a0f49817ac4d1954eaf8e19edf538a7b4e3ef2c24423 quay.io/isovalent/hubble-ui-enterprise:v0.17.0@sha256:e75817a960b04ae012c3e9c0c664fdab51840109524fb721c3b35e3392f14aa5 quay.io/isovalent/operator-generic:v1.11.9-cee.1@sha256:1aaabd1bd3e64053a525c5f89fd7608faa0f6504fd9171944259561445c98d67 quay.io/oauth2-proxy/oauth2-proxy:v7.1.3@sha256:ecd26b74a01f2b547ddaed4d32d35f8f5e09c378d5c1fc6cfa63f0adf659ac2b ``` ### Resources 1.12.3 Manifests: https://docs.isovalent.com/v1.12/public/cilium-ee-olm/cilium-ee-1.12.3.tar.gz Checksum: https://docs.isovalent.com/v1.12/public/cilium-ee-olm/cilium-ee-1.12.3.tar.gz.sha256 Images: ``` quay.io/cilium/certgen:v0.1.8@sha256:4a456552a5f192992a6edcec2febb1c54870d665173a33dc7d876129b199ddbd quay.io/cilium/cilium-etcd-operator:v2.0.7@sha256:04b8327f7f992693c2cb483b999041ed8f92efc8e14f2a5f3ab95574a65ea2dc quay.io/cilium/hubble-export-stdout:v1.0.2@sha256:f1b1475a3c89950481264f5f8ca63eea3b911a45dc83fdf15122d695a0fba9b3 quay.io/cilium/startup-script:d69851597ea019af980891a4628fb36b7880ec26@sha256:c263d1678fb426842c0836358c1da7628d771126211694a3776c4b8500cbb215 quay.io/coreos/etcd:v3.5.4@sha256:795d8660c48c439a7c3764c2330ed9222ab5db5bb524d8d0607cac76f7ba82a3 quay.io/isovalent/cilium:v1.12.3-cee.1@sha256:719d2fa581d8bc8b32eae4f737799f0d42b106eb213edcb53ed0dd88188a594c quay.io/isovalent/clustermesh-apiserver:v1.12.3-cee.1@sha256:765c3d0b4becf17dbbb172ba2f4445b3a52ea2c407c46e032c4cb1d8a7a4e060 quay.io/isovalent/hubble-enterprise-metadata:current@sha256:372511f6fa3523dadfe9128caf77520cf74ce24df4a19e1583ee197375e06014 quay.io/isovalent/hubble-enterprise-operator:v1.8.2@sha256:0d5e6a5d9c07780c18e5aa81d8f1c95c399bff2d36a44fa5f61ac89a788ca83b quay.io/isovalent/hubble-enterprise:v1.8.2@sha256:1e1d3e6c199546b2398f4166ff64210c04075bb407c8f07c5be8d3db08560d4a quay.io/isovalent/hubble-export-fluentd:v1.5.1@sha256:eaf42a84821509b69e2ef2b658796062e79a90e63045a98689da3da99a12e492 quay.io/isovalent/hubble-export-s3:2021-04-20-6f4b384@sha256:ce4f71f7f13301133b7d471b87fb81108b2c2da7fc29865ef96bc8431b2be72b quay.io/isovalent/hubble-rbac:v1.1.0@sha256:9ae9187c25b1d49e9338c6ba4c73be18adee551da2bb9a342602d5060103ae11 quay.io/isovalent/hubble-relay:v1.12.3-cee.1@sha256:03f67eb9e487d007c6992193eb462846bf9476d91dc710d146ff93bd71de0c97 quay.io/isovalent/hubble-ui-enterprise-backend:v0.17.2@sha256:4663af651286050923571bf0ffcd8eb6bfe852af9dc2b9b4c5050a7a23b131e8 quay.io/isovalent/hubble-ui-enterprise:v0.17.2@sha256:498fc2aa10d45f19b18b42ea7814567965b739cfb60a62c4ad6671d3f25db570 quay.io/isovalent/operator-generic:v1.12.3-cee.1@sha256:3262b407f7aa70bf397b4c398848e6204f923ade1ed0512e66461e2e14d76347 quay.io/oauth2-proxy/oauth2-proxy:v7.1.3@sha256:ecd26b74a01f2b547ddaed4d32d35f8f5e09c378d5c1fc6cfa63f0adf659ac2b ``` ### Resources 1.12.6 Manifests: https://docs.isovalent.com/v1.12/public/cilium-ee-olm/cilium-ee-1.12.6.tar.gz Checksum: https://docs.isovalent.com/v1.12/public/cilium-ee-olm/cilium-ee-1.12.6.tar.gz.sha256 Images: ``` quay.io/cilium/certgen:v0.1.8@sha256:4a456552a5f192992a6edcec2febb1c54870d665173a33dc7d876129b199ddbd@sha256:2e0fc99ace29bbe5837718684920112a026bffc46f70efb38ea596c524e68819 quay.io/cilium/cilium-etcd-operator:v2.0.7@sha256:04b8327f7f992693c2cb483b999041ed8f92efc8e14f2a5f3ab95574a65ea2dc@sha256:04b8327f7f992693c2cb483b999041ed8f92efc8e14f2a5f3ab95574a65ea2dc quay.io/cilium/startup-script:d69851597ea019af980891a4628fb36b7880ec26@sha256:c263d1678fb426842c0836358c1da7628d771126211694a3776c4b8500cbb215 quay.io/coreos/etcd:v3.5.4@sha256:795d8660c48c439a7c3764c2330ed9222ab5db5bb524d8d0607cac76f7ba82a3@sha256:a67fb152d4c53223e96e818420c37f11d05c2d92cf62c05ca5604066c37295e9 quay.io/isovalent/cilium:v1.12.6-cee.1@sha256:be14c06a64d3735c547591a0da042d6d63ccde843204b2bfa491b877ac0ac6ab quay.io/isovalent/clustermesh-apiserver:v1.12.6-cee.1@sha256:80045bf094cef749277dcd7a18e7e082d9a049798d9fea3e95e2c40b7f501931 quay.io/isovalent/hubble-enterprise-metadata:current@sha256:372511f6fa3523dadfe9128caf77520cf74ce24df4a19e1583ee197375e06014 quay.io/isovalent/hubble-enterprise-operator:v1.8.5@sha256:216898c8516c436347b94ef528c5a8bbd0f86c0d66b86688a0b56be3f83048f1 quay.io/isovalent/hubble-enterprise:v1.8.5@sha256:e4b05dade11d665f6ae9f6cbb8f203ae185e967d1df0c1fdddb5604a95577ad5 quay.io/isovalent/hubble-export-fluentd:v1.5.1@sha256:eaf42a84821509b69e2ef2b658796062e79a90e63045a98689da3da99a12e492 quay.io/isovalent/hubble-export-s3:2021-04-20-6f4b384@sha256:ce4f71f7f13301133b7d471b87fb81108b2c2da7fc29865ef96bc8431b2be72b quay.io/isovalent/hubble-export-stdout:v1.0.2@sha256:ca42970d2909e57762ac1c2386d1fa74171c181eb83922f0d53ae44f016b3e6d quay.io/isovalent/hubble-rbac:v1.2.0@sha256:0d6da624a03183ea6905aba5bcf12dd9765c60fbc34d861d567de49827b1d2cd quay.io/isovalent/hubble-relay:v1.12.6-cee.1@sha256:26c4e21d908b1aba1e562cffdfd1af7baf88f6f0d6186b1e79aa4f5fc3588916 quay.io/isovalent/hubble-ui-enterprise-backend:v0.17.6@sha256:ede27aecbd2c4d622d1450c91a48ebf01f5606d4751ac4230c409382821d47cf quay.io/isovalent/hubble-ui-enterprise:v0.17.6@sha256:e75323d008bf6bf18c4419266edfa4dc2b98d646924827d9d382fcf59fb682df quay.io/isovalent/operator-generic:v1.12.6-cee.1@sha256:1dbad6ca9134ac06f5f2882a13957b32c9510737c2c8c521234c38a6bbc7cd4e quay.io/oauth2-proxy/oauth2-proxy:v7.1.3@sha256:ecd26b74a01f2b547ddaed4d32d35f8f5e09c378d5c1fc6cfa63f0adf659ac2b ```
Last changed by  
 
Duffie Cooley

Read more

ilt-cilium-foundations-life-of-a-packet-overlay

for the checking of things I think we should consider kubectl exec -ti -n kube-system ds/cilium -- cilium status tho this doesn't show tunnel mode maybe consider custom columns instead of jq. kubectl get nodes -o custom-columns=name:.metadata.name,node-ip:.status.addresses[].address,pod-cidr:.spec.podCIDRs[]

2/15/2023
direct routing lab.

⬢ Cilium Routing Options There are a variety of routing options in Cilium: By default cilium will deploy in an overlay mode and encapsulate all traffic between nodes. We also support Direct Routing. Where we route all traffic between nodes directly using the underlying network In this lab, we will explore what happens to packets that are transmitted between pods when using the Direct Routing option. We don't know that the nodes are part of 10.0.0.0/8 in fact docker exec -ti clab-bgp-cplane-demo-control-plane ip addr shows that they are part of a /24

2/7/2023
ipv6 routed pods with identity.

Link https://hackmd.io/@mauilion/cilium-ipv6 Youtube link https://www.youtube.com/watch?v=IgPqIi67hSA Summary: Setup an cluster and deploy cilium in routed mode. Show connectivity between pods and then use tcpdump to show identity labels encoded in the flow label header for ipv6 traffic. Setup. We will use a kind cluster for this learn more about kind here

12/17/2022
Migrating to cilium v1.11.8 for OpenShift in Kube Proxy Replacement Mode.

Summary of changes. With this release we are deploying a new configmap into the cilium namespace. The name of this configmap is cilium-ee-olm-overrides. The purpose is two fold. the default configmap looks like this: apiVersion: v1 data: RELATED_IMAGE_CERTGEN: quay.io/cilium/certgen:v0.1.5@sha256:0c2b71bb3469990e7990e7e26243617aa344b5a69a4ce465740b8577f9d48ab9@sha256:0c2b71bb3469990e7990e7e26243617aa344b5a69a4ce465740b8577f9d48ab9

9/15/2022
Read more from Duffie Cooley
Published on HackMD