# OpenShift Node Network Configuration
# WORK-IN-PROGRESS

~~~
[jcall@rhdata6 ocp-tacos]$ oc get nncp
NAME STATUS REASON
rhdata1-all-in-one Available SuccessfullyConfigured
rhdata2-all-in-one Available SuccessfullyConfigured
rhdata3-all-in-one Available SuccessfullyConfigured
~~~

~~~
[jcall@rhdata6 ocp-tacos]$ oc get nncp/rhdata1-all-in-one -o yaml
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
annotations:
nmstate.io/webhook-mutating-timestamp: "1668188212258738502"
creationTimestamp: "2022-07-24T22:28:25Z"
generation: 2
name: rhdata1-all-in-one
resourceVersion: "1693703259"
uid: 79e47460-056e-45b2-9ded-d6e16c1b0e9b
spec:
nodeSelector:
kubernetes.io/hostname: rhdata1.dota-lab.iad.redhat.com
desiredState:
dns-resolver:
config:
search:
- dota-lab.iad.redhat.com
server:
- 10.15.168.26
interfaces:
- bridge:
options:
stp:
enabled: false
port:
- name: enp1s0f0
description: Bridge to Red Hat networks and internet (10.15.168.0/24)
ipv4:
address:
- ip: 10.15.168.21
prefix-length: 24
dhcp: false
enabled: true
name: bridge-redhat
state: up
type: linux-bridge
- description: Bridge member (bridge-redhat)
lldp:
enabled: true
name: enp1s0f0
state: up
type: ethernet
- bridge:
options:
stp:
enabled: true
port:
- name: bond-data
description: Bridge to 172.16.1.0/24 and VLAN 999 - 172.31.255.0/24
ipv4:
address:
- ip: 172.16.1.21
prefix-length: 24
dhcp: false
enabled: true
mtu: 9000
name: bridge-data
state: up
type: linux-bridge
- description: LACP bond to arctica-data1 (172.16.1.0/24 and VLAN 999 - 172.31.255.0/24)
ipv4:
enabled: false
ipv6:
enabled: false
link-aggregation:
mode: 802.3ad
options:
lacp_rate: fast
port:
- ens1f0
- ens1f1
lldp:
enabled: true
mtu: 9000
name: bond-data
state: up
type: bond
- description: LACP bond member (bond-data)
lldp:
enabled: true
mtu: 9000
name: ens1f0
state: up
type: ethernet
- description: LACP bond member (bond-data)
lldp:
enabled: true
mtu: 9000
name: ens1f1
state: up
type: ethernet
- bridge:
options:
stp:
enabled: true
port:
- name: bond-priv
description: Bridge to 172.16.2.0/24
ipv4:
address:
- ip: 172.16.2.21
prefix-length: 24
dhcp: false
enabled: true
mtu: 9000
name: bridge-priv
state: up
type: linux-bridge
- description: LACP bond to arctica-data2 (172.16.2.0/24)
ipv4:
enabled: false
ipv6:
enabled: false
link-aggregation:
mode: 802.3ad
options:
lacp_rate: fast
port:
- ens12f0
- ens12f1
lldp:
enabled: true
mtu: 9000
name: bond-priv
state: up
type: bond
- description: LACP bond member (bond-priv)
lldp:
enabled: true
mtu: 9000
name: ens12f0
state: up
type: ethernet
- description: LACP bond member (bond-priv)
lldp:
enabled: true
mtu: 9000
name: ens12f1
state: up
type: ethernet
routes:
config:
- destination: 0.0.0.0/0
next-hop-address: 10.15.168.254
next-hop-interface: bridge-redhat
status:
conditions:
- lastHeartbeatTime: "2023-09-19T11:51:52Z"
lastTransitionTime: "2023-09-19T11:51:52Z"
message: 1/1 nodes successfully configured
reason: SuccessfullyConfigured
status: "True"
type: Available

~~~

## Then create the net-attach-def's

https://access.redhat.com/documentation/en-us/red_hat_openshift_data_foundation/4.13/html-single/managing_and_allocating_storage_resources/index#creating-network-attachment-definitions_rhodf

~~~
[jcall@rhdata6 ocp-tacos]$ oc get net-attach-def -n openshift-storage
NAME AGE
ocs-cluster 299d
ocs-public 299d
[jcall@rhdata6 ocp-tacos]$ oc get net-attach-def/ocs-public -n openshift-storage -o yaml
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
creationTimestamp: "2022-12-01T20:59:08Z"
generation: 1
name: ocs-public
namespace: openshift-storage
resourceVersion: "377237202"
uid: a57c5de3-4e74-4c52-b60a-e541214b04f4
spec:
config: '{ "cniVersion": "0.3.1", "type": "macvlan", "master": "bridge-data", "mode":
"bridge", "ipam": { "type": "whereabouts", "range": "192.168.1.0/24" } }'
[jcall@rhdata6 ocp-tacos]$ oc get net-attach-def/ocs-cluster -n openshift-storage -o yaml
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
creationTimestamp: "2022-12-01T20:59:08Z"
generation: 1
name: ocs-cluster
namespace: openshift-storage
resourceVersion: "377237204"
uid: f5ec9895-e817-4823-a385-e8acc5a8fc6e
spec:
config: '{ "cniVersion": "0.3.1", "type": "macvlan", "master": "bridge-priv", "mode":
"bridge", "ipam": { "type": "whereabouts", "range": "192.168.2.0/24" } }'

~~~

## CLEANUP

apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: cleanup
spec:
  nodeSelector:
    node-role.kubernetes.io/worker: ''
    #kubernetes.io/hostname: node1.example.com    ### Example of targeting 
  desiredState:
    interfaces:
    - name: bridge-rib4
      state: absent
      type: linux-bridge
    - name: bond0
      state: absent
      type: bond

## Phil - 2024-04-30 - Add a VLAN to an interface

The interface (enp5s0f0) is able to use the default/native vlan (921) configured on the switch for OpenShift.
An additional VLAN (923) is available on the same interface (enp5s0f0)
The `NodeNetworkConfigurationPolicies` to create a vlan-interface and assign a static IP address would like like this.
**Please note**: Because static IP addresses are being used, these `NNCPs` must use a `nodeSelector`. Every node needs its own `NNCP` with a unique IP address.

---
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: storage-node-1                          #change
spec:
  nodeSelector:
    kubernetes.io/hostname: node-1.example.com  #change
  desiredState:
    interfaces:
    - description: VLAN 923 (Storage)
      name: enp5s0f0.923                        #match with lines 25-26
      type: vlan
      state: up      
      #mtu: 9000   #default is 1500             #confirm
      ipv4:
        enabled: true
        dhcp: false
        address:
        - ip: 192.1.196.21                      #confirm + change
          prefix-length: 24                     #confirm
      ipv6:
        enabled: false
      vlan:
        base-iface: enp5s0f0                    #confirm
        id: 923                                 #confirm

---
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: storage-node-2                          #change
spec:
  nodeSelector:
    kubernetes.io/hostname: node-2.example.com  #change
  <...snip...>
  
---
---
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: storage-node-3                          #change
spec:
  nodeSelector:
    kubernetes.io/hostname: node-3.example.com  #change
  <...snip...>