--- # System prepended metadata title: Using dedicated networks for OpenShift Data Foundation tags: [network] --- # Using dedicated networks for OpenShift Data Foundation TODO - explain why I'm writing this. In short, I'm writing this in order to provide additional context to the official documentation. [Planning Doc - 8.2 Multi network plug-in (Multus) support](https://docs.redhat.com/en/documentation/red_hat_openshift_data_foundation/4.18/html-single/planning_your_deployment/index#multi-network-plugin-multus-support_rhodf) [Managing Doc - 7 Creating Multus networks](https://docs.redhat.com/en/documentation/red_hat_openshift_data_foundation/4.18/html-single/managing_and_allocating_storage_resources/index#creating-multus-networks_rhodf) TODO - Add a visual diagram that shows overlapping and non-overlapping CIDRs/routes TODO - Add these test commands below. Using these commands you can create Pods/Containers that simulate the OpenShift Data Foundation deployment and verify that you can `ping` between services running on different nodes and also verify that the Pods/Containers can ping the host (and vice-versa) ```bash oc run -n openshift-storage jcall-test-pod --image=registry.redhat.io/rhel9/support-tools --annotations k8s.v1.cni.cncf.io/networks=odf-public -it --restart=Never oc run -n openshift-storage jcall-test-pod --image=registry.redhat.io/rhel9/support-tools --annotations k8s.v1.cni.cncf.io/networks=odf-public -it --restart=Never --command -- ip route list # optionally add a nodeSelector override --overrides='{"spec": {"nodeSelector": {"kubernetes.io/hostname": "dell-r760-01"}}}' ``` :::info Ian mentioned today that it may be possible to simplify the host configuration (NNCP) by setting the netmask to /17 instead of /24. This would eliminiate the need for a static route, when the smaller Node network is a part of the larger Pod network ::: ## Example using bonded NICs instead of a single NIC ### Node configuration (via NodeNetworkConfigurationPolicy) The nodes need to connect with the pods... Make note of the static route that allows the nodes to connect with the pods ```yaml= # Node network: 192.168.255.0/24 (exclude from NetworkAttachmentDefintion's whereabouts) # Pod network: 192.168.128.0/17 (see NetworkAttachmentDefinition) --- apiVersion: nmstate.io/v1 kind: NodeNetworkConfigurationPolicy metadata: name: rhclient1-odf-public-net # [1] spec: nodeSelector: kubernetes.io/hostname: rhclient1.dota-lab.iad.redhat.com # [2] desiredState: interfaces: - name: bridge-data description: Used by VMs (via net-attach-def) to access VLAN 999 (172.31.255.0/24) state: up type: linux-bridge mtu: 9000 bridge: options: stp: enabled: true port: - name: bond-data ipv4: enabled: false ipv6: enabled: false - name: bond-data description: LACP bond to arctica-data1 # 172.16.1.0/24 not used by guacamole # 192.168.128.0/17 used by ODF pods # 192.168.255.0/24 used by ODF nodes # VLAN 999 (172.31.255.0/24) used by VMs state: up type: bond mtu: 9000 link-aggregation: mode: 802.3ad port: - ens2f0 - ens2f1 ipv4: enabled: false ipv6: enabled: false lldp: enabled: true - name: odf-public-shim # [3] description: Used to connect each node to OpenShift Data Foundations' public multus network type: mac-vlan # [4] state: up mtu: 9000 mac-vlan: base-iface: bridge-data # [5] mode: bridge promiscuous: true ipv4: # [6] enabled: true dhcp: false address: - ip: 192.168.255.101 # [7] STATIC IP FOR node-1 prefix-length: 24 routes: # [8] config: - destination: 192.168.128.0/17 # [9] next-hop-interface: odf-public-shim #1. For static IP management, each node must have a unique NodeNetworkConfigurationPolicy. #2. Select individual nodes via hostname to configure static IPs. #3. A “shim” interface is used to connect nodes to the Pods' multus public network. #4. The node’s “shim” interface type must be the same as the Pod's NetworkAttachmentDefinition. #5. The base-iface must match the Multus public network's parent/master interface (typically eth0 or bond0.) #6. The ipv4 (or ipv6`) section configures node IP addresses on the Multus public network. #7. Unique IP address assigned to this node. Don’t forget to change the IP for each node. #8. The routes section instructs nodes how to reach pods on the Multus public network. #9. The route destination(s) of the Pods' CIDR range. ``` ### Pod configuration (via NetworkAttachmentDefinition) https://docs.redhat.com/en/documentation/red_hat_openshift_data_foundation/4.16/html/planning_your_deployment/network-requirements_rhodf#multus-examples_rhodf #### Option 1 - When Node network is a part of the larger Pod network The ODF Pods need to connect with the Nodes. This "just works" if the (smaller CIDR) Node network is within the Pod's network (larger CIDR) **Each Ceph service (mon, osd, etc...) gets an IP address from here** ```yaml= # Node network: 192.168.255.0/24 (exclude from NetworkAttachmentDefintion's whereabouts) # Pod network: 192.168.128.0/17 (see NetworkAttachmentDefinition) --- apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: odf-public namespace: openshift-storage spec: config: '{ "cniVersion": "0.3.1", "type": "macvlan", # [1] "master": "bridge-data", # [2] "mode": "bridge", "mtu": 9000, # Omit unless switch config allows this "ipam": { "type": "whereabouts", # [3] "range": "192.168.128.0/17", # [4] "exclude": [ "192.168.255.0/24" # [5] ] } }' #1. Pods must attach to the parent/master using the same interface type (macvlan) configured via NodeNetworkConfigurationPolicies #2. The net-attach-def's parent/master interface must match the NNCP (typically eth0 or bond0) #3. Using whereabouts instead of DHCP for simplicity. OpenShift will be responsible for address assignment #4. Pods will be assigned IPs in the range 192.168.0.0/16 with the exception of a range allocated to nodes (see 5) #5. Exclude the range assigned to nodes via NNCP #6. The routes section instructs pods how to reach nodes on the Multus public network. A corresponding route should exist in the NNCP #7. The route destination (dst) must match the CIDR range planned for nodes ``` #### Option 2 - When the Node network is not a part of the Pod network But if the node network is outside of the pod CIDR, create a dedicated / static route. I'm not sure why this makes sense, but I had to do it once before... Please note, the Node configuration (NNCP) needs to define a statci route to the Pod network as well *(change #9 of NNCP above)* **Each Ceph service (mon, osd, etc...) gets an IP address from here** ```yaml= # Node network: 192.168.255.0/24 (add static route to Pods' NetworkAttachmentDefintion) # Pod network: 10.252.15.0/24 (add static route to Nodes' NodeNetworkConfigurationPolicies) --- apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: odf-public namespace: openshift-storage spec: config: '{ "cniVersion": "0.3.1", "type": "macvlan", # [1] "master": "bridge-data", # [2] "mode": "bridge", "mtu": 9000, # Omit unless switch config allows this "ipam": { "type": "whereabouts", # [3] "range": "10.252.15.0/24", # [4] "routes": [ # [6] {"dst": "192.168.255.0/24"} # [7] ] } }' #1. Pods must attach to the parent/master using the same interface type (macvlan) configured via NodeNetworkConfigurationPolicies #2. The net-attach-def's parent/master interface must match the NNCP (typically eth0 or bond0) #3. Using whereabouts instead of DHCP for simplicity. OpenShift will be responsible for address assignment #4. Pods will be assigned IPs in the range 192.168.0.0/16 with the exception of a range allocated to nodes (see 5) #5. Exclude the range assigned to nodes via NNCP #6. The routes section instructs pods how to reach nodes on the Multus public network. A corresponding route should exist in the NNCP #7. The route destination (dst) must match the CIDR range planned for nodes ``` # Validation tool Download the tool from here: https://access.redhat.com/articles/7014721 Run the tool like this: ```bash ./rook multus validation config --help ./rook multus validation config converged > my-config vi my-config #add public network, and cluster network if applicable ./rook multus validation run --help ./rook multus validation run --config my-config ``` **Caveat - don't upgrade to 4.17 until the "holder" pods are gone** https://access.redhat.com/articles/7078648 ## Example using single NIC instead of bonded NICs ### Node configuration (via NodeNetworkConfigurationPolicy) Servers need static IP addresses. Only showing control-plane0. Please also create copies (with unique IP addresses) for other nodes. ```yaml= # Node network: 192.168.255.0/24 (exclude from NetworkAttachmentDefintion's whereabouts) # Pod network: 192.168.128.0/17 (see NetworkAttachmentDefinition) --- apiVersion: nmstate.io/v1 kind: NodeNetworkConfigurationPolicy metadata: name: control-plane0-odf-public-shim spec: nodeSelector: kubernetes.io/hostname: control-plane0.example.com ### HOSTNAME desiredState: interfaces: - name: odf-public-shim description: Shim interface used to connect host to OpenShift Data Foundation public Multus network type: mac-vlan state: up mac-vlan: base-iface: enp129s0f1 mode: bridge promiscuous: true ipv4: enabled: true dhcp: false address: - ip: 192.168.255.100 ### STATIC IP prefix-length: 24 - name: enp129s0f1 ## This interface section can be omitted if you don't care about LLDP type: ethernet state: up lldp: enabled: true ## I love LLDP! ipv4: enabled: false ipv6: enabled: false routes: config: - destination: 192.168.128.0/17 next-hop-interface: odf-public-shim ``` ### Pod configuration (via NetworkAttachmentDefinition) **Each Ceph service (mon, osd, etc...) gets an IP address from here** ```yaml= # https://docs.redhat.com/en/documentation/red_hat_openshift_data_foundation/4.16/html/planning_your_deployment/network-requirements_rhodf#multus-examples_rhodf # Node network: 192.168.255.0/24 (exclude from NetworkAttachmentDefintion's whereabouts) # Pod network: 192.168.128.0/17 (see NetworkAttachmentDefinition) # JCALL - the block below doesn't declare any static routes. I assume this is when the node network is within the pod/public/multus network --- apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: odf-public namespace: openshift-storage spec: config: '{ "cniVersion": "0.3.1", "type": "macvlan", # [1] "master": "enp129s0f1", # [2] "mode": "bridge", "ipam": { "type": "whereabouts", # [3] "range": "192.168.128.0/17", # [4] "exclude": [ "192.168.255.0/24" # [5] ] } }' #1. Pods must attach to the parent/master using the same interface type (macvlan) configured via NodeNetworkConfigurationPolicies #2. The net-attach-def's parent/master interface must match the NNCP (typically eth0 or bond0) #3. Using whereabouts instead of DHCP for simplicity. OpenShift will be responsible for address assignment #4. Pods will be assigned IPs in the range 192.168.0.0/16 with the exception of a range allocated to nodes (see 5) #5. Exclude the range assigned to nodes via NNCP #6. The routes section instructs pods how to reach nodes on the Multus public network. A corresponding route should exist in the NNCP #7. The route destination (dst) must match the CIDR range planned for nodes ``` ## Example using VLAN tags (non-overlapping) ### Node configuration (via NodeNetworkConfigurationPolicy) ```yaml= # Node network: 192.168.252.0/24 (add static route to Pods' NetworkAttachmentDefintion) # Pod network: 10.252.16.0/24 (add static route to Nodes' NodeNetworkConfigurationPolicies) --- apiVersion: nmstate.io/v1 kind: NodeNetworkConfigurationPolicy metadata: name: control-plane-0-odf-public-shim-via-vlan-252 spec: nodeSelector: kubernetes.io/hostname: control-plane0.example.com ### HOSTNAME desiredState: interfaces: - name: eth0.252 type: vlan state: up ipv4: enabled: false ipv6: enabled: false lldp: enabled: true vlan: base-iface: eth0 id: 252 protocol: 802.1q - name: odf-public-shim type: mac-vlan state: up ipv4: enabled: true dhcp: false address: - ip: 192.168.252.11 ### Node network prefix-length: 24 ipv6: enabled: false mac-vlan: base-iface: eth0.252 mode: bridge promiscuous: true routes: config: - destination: 10.252.16.0/24 ### Pod network next-hop-interface: odf-public-shim ``` ### Pod configuration (via NetworkAttachmentDefinition) ```yaml= # Node network: 192.168.252.0/24 (add static route to Pods' NetworkAttachmentDefintion) # Pod network: 10.252.16.0/24 (add static route to Nodes' NodeNetworkConfigurationPolicies) --- apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: odf-public namespace: openshift-storage spec: config: '{ "cniVersion": "0.3.1", "type": "macvlan", "master": "eth0.252", "mode": "bridge", "mtu": 9000, ### Omit MTU unless switch config allows this "ipam": { "type": "whereabouts", "range": "10.252.16.0/24", ### Pod network "routes": [ {"dst": "192.168.252.0/24"} ### Node network ] } }' ```