---
# System prepended metadata

title: 設定 Harvester VM 使用獨立的 Bond 網卡
tags: [Harvester]

---

# 設定 Harvester VM 使用獨立的 Bond 網卡

## 1. 新增 Cluster Network 並設定 Bond mode

1. 當前每台 node 額外多出來的網路卡資訊

```
# node 1
$ ip a s
...
4: ens35: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:23:59:d5 brd ff:ff:ff:ff:ff:ff
    altname enp2s3
5: ens36: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:23:59:df brd ff:ff:ff:ff:ff:ff
    altname enp2s4
...

# node 2
$ ip a s
...
4: ens34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:ac:a8:58 brd ff:ff:ff:ff:ff:ff
    altname enp2s2
5: ens35: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:ac:a8:62 brd ff:ff:ff:ff:ff:ff
    altname enp2s3
...

# node 3
$ ip a s
4: ens35: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:f4:48:52 brd ff:ff:ff:ff:ff:ff
    altname enp2s3
5: ens36: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:f4:48:5c brd ff:ff:ff:ff:ff:ff
    altname enp2s4
...
```

2. 建立 `Cluster Network`


左側選單 `Networks` -> `Cluster Network` -> 點選 `Create Cluster Network` 按鈕

![image](https://hackmd.io/_uploads/ryCDsuhsA.png)

取名: `vm` 然後點右下角 `Create` 按鈕


![image](https://hackmd.io/_uploads/r1y4YFniC.png)


點 `Create Network Config` 按鈕

![image](https://hackmd.io/_uploads/Skvrhu3sC.png)

勾選 `Select specific node`
`Node Name`: 點選展開下拉式選單，選擇要設定的主機名稱

![image](https://hackmd.io/_uploads/H1x5AdnoR.png)

點選單 `Uplink`
`NICs` 選擇主機上多的兩張網卡 `ens35` 和 `ens36`
`Bond Options Mode` 選擇 `balance-alb`

![image](https://hackmd.io/_uploads/SkG4ytnsR.png)


再點 `Create Network Config` 按鈕，依照剛剛的步驟將其他節點的網卡都加入到這 Cluster Network，最後就會看到下圖

![image](https://hackmd.io/_uploads/SJG_Yt2sR.png)

> 要確認每個 `Config Network` 狀態都是 `Active`

## 2. 永久設定 brige 裝置一個 IP 位置 (可選)

:::info
讓 Harvester node 能夠連接使用這個 Network 的 VM，如果不做此步驟則 VM 建出來會無法從 Harvester node 連接，但還是能夠從同網段的機器連到 VM。
:::


### 2.1. 檢查當前網卡的設定

```
$ ip a s
4: ens35: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vm-bo state UP group default qlen 1000
    link/ether 00:0c:29:23:59:d5 brd ff:ff:ff:ff:ff:ff
    altname enp2s3
5: ens36: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vm-bo state UP group default qlen 1000
    link/ether 00:0c:29:23:59:df brd ff:ff:ff:ff:ff:ff
    altname enp2s4
...
65: vm-bo: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master vm-br state UP group default qlen 1000
    link/ether 00:0c:29:23:59:d5 brd ff:ff:ff:ff:ff:ff
66: vm-br: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
    link/ether 1e:df:20:6d:88:47 brd ff:ff:ff:ff:ff:ff
```

> 發現多了 `vm-bo` 這張網卡和 `vm-br` 這個 bridge 裝置
> 但是 `vm-br` 橋接裝置沒有自己的 IP

### 2.2. 備份 Harvester Configuration 檔案 `/oem/90_custom.yaml`

```
$ sudo cp /oem/90_custom.yaml /oem/install/90_custom.yaml.$(date --iso-8601=minutes)
```

### 2.3. 修改 `/oem/90_custom.yaml` 檔案，並且更新以下 Yaml 檔中路徑的值 `stages.initramfs[0].files`

```
$ vi /oem/90_custom.yaml
```

內容如下

```
name: Harvester Configuration
stages:
    initramfs:
        - commands:
              ...
          files:
          # add the following context
            - path: /etc/wicked/scripts/setup_vm_bond.sh
              permissions: 493
              owner: 0
              group: 0
              content: |
                #!/bin/sh

                ACTION=$1
                INTERFACE=$2

                case $ACTION in
                        post-up)
                                # inherit MAC address
                                ip link set dev vm-br address $(ip -json link show dev $INTERFACE | jq -j '.[0]["address"]')

                                # accept all vlan, PVID=1 by default
                                bridge vlan add vid 2-4094 dev $INTERFACE
                                ;;

                esac
              encoding: ""
              ownerstring: ""
            - path: /etc/sysconfig/network/ifcfg-vm-bo
              permissions: 384
              owner: 0
              group: 0
              content: |+
                STARTMODE='onboot'
                BONDING_MASTER='yes'
                BOOTPROTO='none'
                POST_UP_SCRIPT="wicked:setup_vm_bond.sh"


                BONDING_SLAVE_0='ens35'
                BONDING_SLAVE_1='ens36'

                BONDING_MODULE_OPTS='miimon=100 mode=balance-alb '

                DHCLIENT_SET_DEFAULT_ROUTE='no'

              encoding: ""
            - path: /etc/sysconfig/network/ifcfg-ens35
              permissions: 384
              owner: 0
              group: 0
              content: |
                STARTMODE='hotplug'
                BOOTPROTO='none'
              encoding: ""
              ownerstring: ""
            - path: /etc/sysconfig/network/ifcfg-ens36
              permissions: 384
              owner: 0
              group: 0
              content: |
                STARTMODE='hotplug'
                BOOTPROTO='none'
              encoding: ""
              ownerstring: ""
            - path: /etc/wicked/scripts/setup_vm_bridge.sh
              permissions: 493
              owner: 0
              group: 0
              content: |
                #!/bin/sh

                ACTION=$1
                INTERFACE=$2

                case $ACTION in
                        pre-up)
                                # enable vlan-aware
                                ip link set $INTERFACE type bridge vlan_filtering 1
                                ;;

                        post-up)
                                # accept all vlan, PVID=1 by default
                                bridge vlan add vid 2-4094 dev $INTERFACE self
                                bridge vlan add vid 2-4094 dev vm-bo
                                ;;
                esac
              encoding: ""
              ownerstring: ""
            - path: /etc/sysconfig/network/ifcfg-vm-br
              permissions: 384
              owner: 0
              group: 0
              content: |+
                STARTMODE='onboot'
                BOOTPROTO='static'
                BRIDGE='yes'
                BRIDGE_STP='off'
                BRIDGE_FORWARDDELAY='0'
                BRIDGE_PORTS='vm-bo'
                PRE_UP_SCRIPT="wicked:setup_vm_bridge.sh"
                POST_UP_SCRIPT="wicked:setup_vm_bridge.sh"

                IPADDR=192.168.81.21
                NETMASK=255.255.255.0


                DHCLIENT_SET_DEFAULT_ROUTE='yes'


              encoding: ""
              ownerstring: ""
```

1. `/etc/wicked/scripts/setup_vm_bond.sh` 修改內容如下 :
    -  `ip link set dev <你的 bridge 名稱> address`
        - bridge 的名稱會是 `<cluster network name>-br`
        - 譬如: 我的 `cluster network` 取名為 `vm`，那麼名稱就是 `vm-br`
    - 此檔名可自訂義
2. `/etc/sysconfig/network/ifcfg-vm-bo` 修改內容如下 :
    -  `POST_UP_SCRIPT="wicked:setup_vm_bond.sh"`
        - 格式 : `wicked:<第一個 script 的名稱>`
    - `BONDING_SLAVE_0='ens35'`
        - 格式 : `ONDING_SLAVE_0='第一張網卡名稱'`
    - `BONDING_SLAVE_1='ens36'`
        - 格式 : `ONDING_SLAVE_1='第二張網卡名稱'`
    - `BONDING_MODULE_OPTS='miimon=100 mode=balance-alb '`
        - 格式 : `BONDING_MODULE_OPTS='miimon=<在 UI 上建立的值> node=<在 UI 上選的 Bond mode>'`
3. `/etc/sysconfig/network/ifcfg-ens35` 修改內容如下 :
    - 檔名須為: `ifcfg-<網卡名稱>`
4. `/etc/sysconfig/network/ifcfg-ens36` 修改內容如下 :
    - 檔名須為: `ifcfg-<網卡名稱>`
5. `/etc/wicked/scripts/setup_vm_bridge.sh` 修改內容如下 :
    - 檔名可自訂義
6. `/etc/sysconfig/network/ifcfg-vm-br` 修改內容如下 : 
    - `BRIDGE_PORTS='vm-bo'`
        - 格式: `BRIDGE_PORTS='bridge 名稱'`
    - `PRE_UP_SCRIPT="wicked:setup_vm_bridge.sh"`
        - 格式: `PRE_UP_SCRIPT="上一個 Script 檔案的名稱"`
    - `POST_UP_SCRIPT="wicked:setup_vm_bridge.sh"`
        - 格式: `POST_UP_SCRIPT="上一個 Script 檔案的名稱"`
    - `IPADDR=192.168.81.21`
        - 改成符合這張網卡的 IP Address
    - `NETMASK=255.255.255.0`
        - 改成符合這張網卡的子網路遮罩

### 2.4. 重開機

```
$ sudo init 6
```

> 如果這台 node 上有正在跑的 VM 要先關機。


### 2.5. 確認成功幫 Bridge 設靜態 IP


```
$ ip a s
...
4: ens35: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vm-bo state UP group default qlen 1000
    link/ether 00:0c:29:f4:48:52 brd ff:ff:ff:ff:ff:ff
    altname enp2s3
5: ens36: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vm-bo state UP group default qlen 1000
    link/ether 00:0c:29:f4:48:5c brd ff:ff:ff:ff:ff:ff
    altname enp2s4
6: vm-br: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:0c:29:f4:48:52 brd ff:ff:ff:ff:ff:ff
    inet 192.168.81.23/24 brd 192.168.81.255 scope global vm-br
       valid_lft forever preferred_lft forever
7: vm-bo: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master vm-br state UP group default qlen 1000
    link/ether 00:0c:29:f4:48:52 brd ff:ff:ff:ff:ff:ff
```

### 2.6. 重複以上 2.1. ~ 2.5. 的步驟依序做完每一台有在 `Cluster Netwrok` 底下的節點

這樣設定完 Harvester node 就能夠連到 VM


## 3. 建立 VM Network


點選左側選單 Networks -> VM Networks -> Create

![image](https://hackmd.io/_uploads/Syki2Fhi0.png)


* **Name**: `vm`
* **Type**: `UntaggedNetwrok`
* **Cluster Network**: `vm`


![image](https://hackmd.io/_uploads/rk0l3YhiA.png)

建立成功，之前 VM 在建立的時候就能有自己獨立的網卡辣~


![image](https://hackmd.io/_uploads/BJVP2YnsC.png)
