# JD working session

More info at https://hackmd.io/@johnsimcall/H1DCTW230

## OpenShift Data Foundations via dedicated 10Gb

### Each Ceph service (mon, osd, etc...) gets an IP address from here

# https://docs.redhat.com/en/documentation/red_hat_openshift_data_foundation/4.16/html/planning_your_deployment/network-requirements_rhodf#multus-examples_rhodf

# Node network: 192.168.255.0/24  (exclude from NetworkAttachmentDefintion's whereabouts)
# Pod  network: 192.168.128.0/17  (see NetworkAttachmentDefinition)

# JCALL - the block below doesn't declare any static routes. I assume this is when the node network is within the pod/public/multus network
---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
 name: odf-public
 namespace: openshift-storage
spec:
 config: '{
     "cniVersion": "0.3.1",
     "type": "macvlan",               # [1]
     "master": "enp129s0f1",          # [2]
     "mode": "bridge",
     "ipam": {
       "type": "whereabouts",         # [3]
       "range": "192.168.128.0/17",   # [4]
       "exclude": [
          "192.168.255.0/24"          # [5]
       ]
     }
   }'

#1. Pods must attach to the parent/master using the same interface type (macvlan) configured via NodeNetworkConfigurationPolicies
#2. The net-attach-def's parent/master interface must match the NNCP (typically eth0 or bond0)
#3. Using whereabouts instead of DHCP for simplicity. OpenShift will be responsible for address assignment
#4. Pods will be assigned IPs in the range 192.168.0.0/16 with the exception of a range allocated to nodes (see 5)
#5. Exclude the range assigned to nodes via NNCP
#6. The routes section instructs pods how to reach nodes on the Multus public network. A corresponding route should exist in the NNCP
#7. The route destination (dst) must match the CIDR range planned for nodes

### Servers need static IP addresses too

#### control-plane0

# Node network: 192.168.255.0/24  (exclude from NetworkAttachmentDefintion's whereabouts)
# Pod  network: 192.168.128.0/17  (see NetworkAttachmentDefinition)

---
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
 name: control-plane0-odf-public-shim
spec:
 nodeSelector:
   kubernetes.io/hostname: control-plane0.example.com  ### HOSTNAME
 desiredState:
   interfaces:
     - name: odf-public-shim
       description: Shim interface used to connect host to OpenShift Data Foundation public Multus network
       type: mac-vlan
       state: up
       mac-vlan:
         base-iface: enp129s0f1
         mode: bridge
         promiscuous: true
       ipv4:
         enabled: true
         dhcp: false
         address:
           - ip: 192.168.255.100  ### STATIC IP
             prefix-length: 24

     - name: enp129s0f1
       type: ethernet
       state: up
       lldp:
         enabled: true
       ipv4:
         enabled: false
       ipv6:
         enabled: false

   routes:
     config:
       - destination: 192.168.128.0/17
         next-hop-interface: odf-public-shim

#### control-plane1

# Node network: 192.168.255.0/24  (exclude from NetworkAttachmentDefintion's whereabouts)
# Pod  network: 192.168.128.0/17  (see NetworkAttachmentDefinition)

---
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
 name: control-plane1-odf-public-shim
spec:
 nodeSelector:
   kubernetes.io/hostname: control-plane1.example.com  ### HOSTNAME
 desiredState:
   interfaces:
     - name: odf-public-shim
       description: Shim interface used to connect host to OpenShift Data Foundation public Multus network
       type: mac-vlan
       state: up
       mac-vlan:
         base-iface: enp129s0f1
         mode: bridge
         promiscuous: true
       ipv4:
         enabled: true
         dhcp: false
         address:
           - ip: 192.168.255.101  ### STATIC IP
             prefix-length: 24
             
     - name: enp129s0f1
       type: ethernet
       state: up
       lldp:
         enabled: true
       ipv4:
         enabled: false
       ipv6:
         enabled: false

   routes:
     config:
       - destination: 192.168.128.0/17
         next-hop-interface: odf-public-shim

#### control-plane2

# Node network: 192.168.255.0/24  (exclude from NetworkAttachmentDefintion's whereabouts)
# Pod  network: 192.168.128.0/17  (see NetworkAttachmentDefinition)

---
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
 name: control-plane2-odf-public-shim
spec:
 nodeSelector:
   kubernetes.io/hostname: control-plane2.example.com  ### HOSTNAME
 desiredState:
   interfaces:
     - name: odf-public-shim
       description: Shim interface used to connect host to OpenShift Data Foundation public Multus network
       type: mac-vlan
       state: up
       mac-vlan:
         base-iface: enp129s0f1
         mode: bridge
         promiscuous: true
       ipv4:
         enabled: true
         dhcp: false
         address:
           - ip: 192.168.255.102  ### STATIC IP
             prefix-length: 24

     - name: enp129s0f1
       type: ethernet
       state: up
       lldp:
         enabled: true
       ipv4:
         enabled: false
       ipv6:
         enabled: false

   routes:
     config:
       - destination: 192.168.128.0/17
         next-hop-interface: odf-public-shim

### Cleanup host networking mistakes

We used this procedure to cleanup the nodes' networking after my typos.

oc delete nncp control-plane0-odf-public-shim control-plane1-odf-public-shim control-plane2-odf-public-shim

---
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
 name: cleanup-odf-public-shim
spec:
 nodeSelector:
   node-role.kubernetes.io/master: ''
 desiredState:
   interfaces:
     - name: odf-public-shim
       state: absent

oc delete nncp cleanup-odf-public-shim

## Validation tool

Download the tool from here: https://access.redhat.com/articles/7014721

Run the tool like this:

./rook multus validation config --help
./rook multus validation config converged > my-config
vi my-config  #add public network, and cluster network if applicable

./rook multus validation run --help
./rook multus validation run --config my-config

Re-run the tool

oc delete configmap/multus-validation-test-owner