# OpenShift networking for iSCSI redundancy

I recently spoke with a friend who wanted their OpenShift Virtualization hypervisors / OpenShift nodes to access their iSCSI strorage array ([Dell PowerStore](https://dell.github.io/csm-docs/docs/getting-started/installation/openshift/powerstore/csmoperator/)) using two VLAN interfaces for multipath redundancy.

:::info
I consolidated Dell's 3x MachineConfigs for iSCSI + multipathd into one. ([GitHub Gist](https://gist.github.com/johnsimcall/e546204290949bdab375e767b104512a))
:::

Unfortunately the web-based / GUI interface for creating node-level networking configurations doesn't support VLAN interfaces yet, but it can be done with YAML. The `NodeNetworkConfigurationPolicy` below accomplished the task.

The YAML below configures two NICs (`eno12419` + `eno12429`) into an LACP active/active bond called `bond1`. Please note, `bond0` was created when OpenShift was installed. The external switch trunks multiple VLANs to the bonded interfaces. Two VLAN interfaces are created, one per bond, with static IP addresses assigned. An extra OVS bridge is created and connected to `bond1` for the VMs running on OpenShift.

:::warning
The static IP address configuration of the VLAN interfaces doesn't add any extra routes/gateway or DNS information. In other words, the iSCSI networks are private isolated networks.
:::

:::info
I find it helpful to replace the term `bridge` with `virtual switch` when I think about this. In technical terms, a `switch` is also known as a`multiport bridge`
- *"A network switch is also called a MAC bridge by the [IEEE](https://en.wikipedia.org/wiki/IEEE_Standards_Association)"* - [Wikipedia: Network switch](https://en.wikipedia.org/wiki/Network_switch)
- *"The multiport bridge function serves as the basis for network switches"* - [Wikipedia: Network bridge](https://en.wikipedia.org/wiki/Network_bridge)
:::

## Bad stick-figure diagram

### idea #2

	bond0.1341				br-bond1.1342 (13 chars)
	|					|
	bond0					bond1
eno12399	eno12409		eno12419	eno12429

## Node Networking template
Use this `NodeNetworkConfigurationPolicy` as a template for creating bonds and vlan-interfaces with static IP addressing.

---
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: server01-iscsi
spec:
  nodeSelector:
    kubernetes.io/hostname: 'server01.example.com'  ### CHANGE THIS
  desiredState:
    interfaces:
    - name: bond0.1341       ###
      type: vlan
      state: up
      vlan:
        id: 1341             ###
        base-iface: bond0    ###
      ipv4:
        enabled: true
        dhcp: false
        address:
        - ip: 10.130.41.101   ###
          prefix-length: 24   ###
      ipv6:
        enabled: false


    - name: bond1.1342        ###
      type: vlan
      state: up
      vlan:
        id: 1342              ###
        base-iface: bond1     ###
      ipv4:
        enabled: true
        dhcp: false
        address:
        - ip: 10.130.42.101   ###
          prefix-length: 24   ###
      ipv6:
        enabled: false


    - name: bond1
      type: bond
      state: up
      controller: bond1-br
      link-aggregation:
        mode: 802.3ad         ###
        port:
        - eno12419            ###
        - eno12429            ###
      ipv4:
        enabled: false
      ipv6:
        enabled: false
        

    - name: bond1-br
      profile-name: bond1-br
      type: ovs-bridge
      state: up
      bridge:
        port:
        - name: bond1
      ipv4:
        enabled: false
      ipv6:
        enabled: false


    ovn:
      bridge-mappings:
      - localnet: trunk
        bridge: bond1-br
        state: present

## Creating the VM networks (vSwitch)

---
apiVersion: k8s.ovn.org/v1
kind: ClusterUserDefinedNetwork
metadata:
  name: vlan-4
spec:
  namespaceSelector:
    matchExpressions:
      - key: kubernetes.io/metadata.name
        operator: In
        values:
          - default
  network:
    topology: Localnet
    localnet:
      physicalNetworkName: trunk
      role: Secondary
      ipam:
        mode: Disabled
      vlan:
        mode: Access
        access:
          id: 4

## Kudos / Acknowledments

I found the [nmstate examples](https://nmstate.io/devel/yaml_api.html#openvswitch-bridge-interface) to be very helpful.

A [complete example that includes the `NodeNetworkConfiguration` pieces](https://nmstate.io/kubernetes-nmstate/examples.html#open-vswitch-bridge-interface) was also very helpful!

## Appendix

For those that may be curious, the configuration above looks like this when you run `nmcli connection show`

[root@rhel9 ~]# nmcli con show
NAME                   UUID                                  TYPE           DEVICE