# Using dedicated networks for OpenShift Data Foundation

TODO - explain why I'm writing this. In short, I'm writing this in order to provide additional context to the official documentation.

[Planning Doc - 8.2 Multi network plug-in (Multus) support](https://docs.redhat.com/en/documentation/red_hat_openshift_data_foundation/4.18/html-single/planning_your_deployment/index#multi-network-plugin-multus-support_rhodf)
[Managing Doc - 7 Creating Multus networks](https://docs.redhat.com/en/documentation/red_hat_openshift_data_foundation/4.18/html-single/managing_and_allocating_storage_resources/index#creating-multus-networks_rhodf)

TODO - Add a visual diagram that shows overlapping and non-overlapping CIDRs/routes

TODO - Add these test commands below. Using these commands you can create Pods/Containers that simulate the OpenShift Data Foundation deployment and verify that you can `ping` between services running on different nodes and also verify that the Pods/Containers can ping the host (and vice-versa)

oc run -n openshift-storage jcall-test-pod --image=registry.redhat.io/rhel9/support-tools --annotations k8s.v1.cni.cncf.io/networks=odf-public -it --restart=Never 

oc run -n openshift-storage jcall-test-pod --image=registry.redhat.io/rhel9/support-tools --annotations k8s.v1.cni.cncf.io/networks=odf-public -it --restart=Never --command -- ip route list

# optionally add a nodeSelector override
 --overrides='{"spec": {"nodeSelector": {"kubernetes.io/hostname": "dell-r760-01"}}}'

:::info
Ian mentioned today that it may be possible to simplify the host configuration (NNCP) by setting the netmask to /17 instead of /24. This would eliminiate the need for a static route, when the smaller Node network is a part of the larger Pod network
:::

## Example using bonded NICs instead of a single NIC

### Node configuration (via NodeNetworkConfigurationPolicy)

The nodes need to connect with the pods...
Make note of the static route that allows the nodes to connect with the pods

# Node network: 192.168.255.0/24  (exclude from NetworkAttachmentDefintion's whereabouts)
# Pod  network: 192.168.128.0/17  (see NetworkAttachmentDefinition)
---
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: rhclient1-odf-public-net  # [1]
spec:
  nodeSelector:
    kubernetes.io/hostname: rhclient1.dota-lab.iad.redhat.com  # [2]
  desiredState:
    interfaces:
    - name: bridge-data
      description: Used by VMs (via net-attach-def) to access VLAN 999 (172.31.255.0/24)
      state: up
      type: linux-bridge
      mtu: 9000
      bridge:
        options:
          stp:
            enabled: true
        port:
        - name: bond-data
      ipv4:
        enabled: false
      ipv6:
        enabled: false

    - name: bond-data
      description: LACP bond to arctica-data1
      # 172.16.1.0/24    not used by guacamole
      # 192.168.128.0/17 used by ODF pods
      # 192.168.255.0/24 used by ODF nodes
      # VLAN 999 (172.31.255.0/24) used by VMs
      state: up
      type: bond
      mtu: 9000
      link-aggregation:
        mode: 802.3ad
        port:
        - ens2f0
        - ens2f1
      ipv4:
        enabled: false
      ipv6:
        enabled: false
      lldp:
        enabled: true

    - name: odf-public-shim  # [3]
      description: Used to connect each node to OpenShift Data Foundations' public multus network
      type: mac-vlan  # [4]
      state: up
      mtu: 9000
      mac-vlan:
        base-iface: bridge-data  # [5]
        mode: bridge
        promiscuous: true
      ipv4:  # [6]
        enabled: true
        dhcp: false
        address:
        - ip: 192.168.255.101  # [7] STATIC IP FOR node-1
          prefix-length: 24

    routes:  # [8]
      config:
      - destination: 192.168.128.0/17  # [9]
        next-hop-interface: odf-public-shim

#1. For static IP management, each node must have a unique NodeNetworkConfigurationPolicy.
#2. Select individual nodes via hostname to configure static IPs.
#3. A “shim” interface is used to connect nodes to the Pods' multus public network.
#4. The node’s “shim” interface type must be the same as the Pod's NetworkAttachmentDefinition.
#5. The base-iface must match the Multus public network's parent/master interface (typically eth0 or bond0.)
#6. The ipv4 (or ipv6`) section configures node IP addresses on the Multus public network.
#7. Unique IP address assigned to this node. Don’t forget to change the IP for each node.
#8. The routes section instructs nodes how to reach pods on the Multus public network.
#9. The route destination(s) of the Pods' CIDR range.

### Pod configuration (via NetworkAttachmentDefinition)

https://docs.redhat.com/en/documentation/red_hat_openshift_data_foundation/4.16/html/planning_your_deployment/network-requirements_rhodf#multus-examples_rhodf

#### Option 1 - When Node network is a part of the larger Pod network

The ODF Pods need to connect with the Nodes. This "just works" if the (smaller CIDR) Node network is within the Pod's network (larger CIDR)
**Each Ceph service (mon, osd, etc...) gets an IP address from here**

# Node network: 192.168.255.0/24  (exclude from NetworkAttachmentDefintion's whereabouts)
# Pod  network: 192.168.128.0/17  (see NetworkAttachmentDefinition)

---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
 name: odf-public
 namespace: openshift-storage
spec:
 config: '{
     "cniVersion": "0.3.1",
     "type": "macvlan",               # [1]
     "master": "bridge-data",         # [2]
     "mode": "bridge",
     "mtu": 9000,                     # Omit unless switch config allows this
     "ipam": {
       "type": "whereabouts",         # [3]
       "range": "192.168.128.0/17",   # [4]
       "exclude": [
          "192.168.255.0/24"          # [5]
       ]
     }
   }'

#1. Pods must attach to the parent/master using the same interface type (macvlan) configured via NodeNetworkConfigurationPolicies
#2. The net-attach-def's parent/master interface must match the NNCP (typically eth0 or bond0)
#3. Using whereabouts instead of DHCP for simplicity. OpenShift will be responsible for address assignment
#4. Pods will be assigned IPs in the range 192.168.0.0/16 with the exception of a range allocated to nodes (see 5)
#5. Exclude the range assigned to nodes via NNCP
#6. The routes section instructs pods how to reach nodes on the Multus public network. A corresponding route should exist in the NNCP
#7. The route destination (dst) must match the CIDR range planned for nodes

#### Option 2 - When the Node network is not a part of the Pod network

But if the node network is outside of the pod CIDR, create a dedicated / static route.
I'm not sure why this makes sense, but I had to do it once before...
Please note, the Node configuration (NNCP) needs to define a statci route to the Pod network as well *(change #9 of NNCP above)*
**Each Ceph service (mon, osd, etc...) gets an IP address from here**

# Node network: 192.168.255.0/24  (add static route to Pods' NetworkAttachmentDefintion)
# Pod  network:   10.252.15.0/24  (add static route to Nodes' NodeNetworkConfigurationPolicies)

---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
 name: odf-public
 namespace: openshift-storage
spec:
 config: '{
     "cniVersion": "0.3.1",
     "type": "macvlan",               # [1]
     "master": "bridge-data",         # [2]
     "mode": "bridge",
     "mtu": 9000,                     # Omit unless switch config allows this
     "ipam": {
       "type": "whereabouts",         # [3]
       "range": "10.252.15.0/24",     # [4]
       "routes": [                    # [6]
         {"dst": "192.168.255.0/24"}  # [7]
       ]
     }
   }'

#1. Pods must attach to the parent/master using the same interface type (macvlan) configured via NodeNetworkConfigurationPolicies
#2. The net-attach-def's parent/master interface must match the NNCP (typically eth0 or bond0)
#3. Using whereabouts instead of DHCP for simplicity. OpenShift will be responsible for address assignment
#4. Pods will be assigned IPs in the range 192.168.0.0/16 with the exception of a range allocated to nodes (see 5)
#5. Exclude the range assigned to nodes via NNCP
#6. The routes section instructs pods how to reach nodes on the Multus public network. A corresponding route should exist in the NNCP
#7. The route destination (dst) must match the CIDR range planned for nodes

# Validation tool

Download the tool from here: https://access.redhat.com/articles/7014721

Run the tool like this:

./rook multus validation config --help
./rook multus validation config converged > my-config
vi my-config  #add public network, and cluster network if applicable

./rook multus validation run --help
./rook multus validation run --config my-config

**Caveat - don't upgrade to 4.17 until the "holder" pods are gone**
https://access.redhat.com/articles/7078648

## Example using single NIC instead of bonded NICs

### Node configuration (via NodeNetworkConfigurationPolicy)

Servers need static IP addresses. Only showing control-plane0. Please also create copies (with unique IP addresses) for other nodes.

# Node network: 192.168.255.0/24  (exclude from NetworkAttachmentDefintion's whereabouts)
# Pod  network: 192.168.128.0/17  (see NetworkAttachmentDefinition)

---
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
 name: control-plane0-odf-public-shim
spec:
 nodeSelector:
   kubernetes.io/hostname: control-plane0.example.com  ### HOSTNAME
 desiredState:
   interfaces:
     - name: odf-public-shim
       description: Shim interface used to connect host to OpenShift Data Foundation public Multus network
       type: mac-vlan
       state: up
       mac-vlan:
         base-iface: enp129s0f1
         mode: bridge
         promiscuous: true
       ipv4:
         enabled: true
         dhcp: false
         address:
           - ip: 192.168.255.100  ### STATIC IP
             prefix-length: 24

     - name: enp129s0f1  ## This interface section can be omitted if you don't care about LLDP
       type: ethernet
       state: up
       lldp:
         enabled: true   ## I love LLDP!
       ipv4:
         enabled: false
       ipv6:
         enabled: false

   routes:
     config:
       - destination: 192.168.128.0/17
         next-hop-interface: odf-public-shim

### Pod configuration (via NetworkAttachmentDefinition)

**Each Ceph service (mon, osd, etc...) gets an IP address from here**

# https://docs.redhat.com/en/documentation/red_hat_openshift_data_foundation/4.16/html/planning_your_deployment/network-requirements_rhodf#multus-examples_rhodf

# Node network: 192.168.255.0/24  (exclude from NetworkAttachmentDefintion's whereabouts)
# Pod  network: 192.168.128.0/17  (see NetworkAttachmentDefinition)

# JCALL - the block below doesn't declare any static routes. I assume this is when the node network is within the pod/public/multus network
---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
 name: odf-public
 namespace: openshift-storage
spec:
 config: '{
     "cniVersion": "0.3.1",
     "type": "macvlan",               # [1]
     "master": "enp129s0f1",          # [2]
     "mode": "bridge",
     "ipam": {
       "type": "whereabouts",         # [3]
       "range": "192.168.128.0/17",   # [4]
       "exclude": [
          "192.168.255.0/24"          # [5]
       ]
     }
   }'

#1. Pods must attach to the parent/master using the same interface type (macvlan) configured via NodeNetworkConfigurationPolicies
#2. The net-attach-def's parent/master interface must match the NNCP (typically eth0 or bond0)
#3. Using whereabouts instead of DHCP for simplicity. OpenShift will be responsible for address assignment
#4. Pods will be assigned IPs in the range 192.168.0.0/16 with the exception of a range allocated to nodes (see 5)
#5. Exclude the range assigned to nodes via NNCP
#6. The routes section instructs pods how to reach nodes on the Multus public network. A corresponding route should exist in the NNCP
#7. The route destination (dst) must match the CIDR range planned for nodes

## Example using VLAN tags (non-overlapping)

### Node configuration (via NodeNetworkConfigurationPolicy)

# Node network: 192.168.252.0/24  (add static route to Pods' NetworkAttachmentDefintion)
# Pod  network:   10.252.16.0/24  (add static route to Nodes' NodeNetworkConfigurationPolicies)

---
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
 name: control-plane-0-odf-public-shim-via-vlan-252
spec:
 nodeSelector:
   kubernetes.io/hostname: control-plane0.example.com  ### HOSTNAME
 desiredState:
   interfaces:
   - name: eth0.252
     type: vlan
     state: up
     ipv4:
       enabled: false
     ipv6:
       enabled: false
     lldp:
       enabled: true
     vlan:
       base-iface: eth0
       id: 252
       protocol: 802.1q
       
   - name: odf-public-shim
     type: mac-vlan
     state: up
     ipv4:
       enabled: true
       dhcp: false
       address:
       - ip: 192.168.252.11   ### Node network
         prefix-length: 24
     ipv6:
       enabled: false
     mac-vlan:
       base-iface: eth0.252
       mode: bridge
       promiscuous: true
       
   routes:
     config:
     - destination: 10.252.16.0/24    ### Pod network
       next-hop-interface: odf-public-shim

### Pod configuration (via NetworkAttachmentDefinition)

# Node network: 192.168.252.0/24  (add static route to Pods' NetworkAttachmentDefintion)
# Pod  network:   10.252.16.0/24  (add static route to Nodes' NodeNetworkConfigurationPolicies)

---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
 name: odf-public
 namespace: openshift-storage
spec:
 config: '{
     "cniVersion": "0.3.1",
     "type": "macvlan",
     "master": "eth0.252",
     "mode": "bridge",
     "mtu": 9000,       ### Omit MTU unless switch config allows this
     "ipam": {
       "type": "whereabouts",
       "range": "10.252.16.0/24",     ### Pod network
       "routes": [
         {"dst": "192.168.252.0/24"}  ### Node network
       ]
     }
   }'