# 設定 Harvester VM 使用獨立的 Bond 網卡 ## 1. 新增 Cluster Network 並設定 Bond mode 1. 當前每台 node 額外多出來的網路卡資訊 ``` # node 1 $ ip a s ... 4: ens35: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:23:59:d5 brd ff:ff:ff:ff:ff:ff altname enp2s3 5: ens36: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:23:59:df brd ff:ff:ff:ff:ff:ff altname enp2s4 ... # node 2 $ ip a s ... 4: ens34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:ac:a8:58 brd ff:ff:ff:ff:ff:ff altname enp2s2 5: ens35: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:ac:a8:62 brd ff:ff:ff:ff:ff:ff altname enp2s3 ... # node 3 $ ip a s 4: ens35: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:f4:48:52 brd ff:ff:ff:ff:ff:ff altname enp2s3 5: ens36: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:f4:48:5c brd ff:ff:ff:ff:ff:ff altname enp2s4 ... ``` 2. 建立 `Cluster Network` 左側選單 `Networks` -> `Cluster Network` -> 點選 `Create Cluster Network` 按鈕 ![image](https://hackmd.io/_uploads/ryCDsuhsA.png) 取名: `vm` 然後點右下角 `Create` 按鈕 ![image](https://hackmd.io/_uploads/r1y4YFniC.png) 點 `Create Network Config` 按鈕 ![image](https://hackmd.io/_uploads/Skvrhu3sC.png) 勾選 `Select specific node` `Node Name`: 點選展開下拉式選單，選擇要設定的主機名稱 ![image](https://hackmd.io/_uploads/H1x5AdnoR.png) 點選單 `Uplink` `NICs` 選擇主機上多的兩張網卡 `ens35` 和 `ens36` `Bond Options Mode` 選擇 `balance-alb` ![image](https://hackmd.io/_uploads/SkG4ytnsR.png) 再點 `Create Network Config` 按鈕，依照剛剛的步驟將其他節點的網卡都加入到這 Cluster Network，最後就會看到下圖 ![image](https://hackmd.io/_uploads/SJG_Yt2sR.png) > 要確認每個 `Config Network` 狀態都是 `Active` ## 2. 永久設定 brige 裝置一個 IP 位置 (可選) :::info 讓 Harvester node 能夠連接使用這個 Network 的 VM，如果不做此步驟則 VM 建出來會無法從 Harvester node 連接，但還是能夠從同網段的機器連到 VM。 ::: ### 2.1. 檢查當前網卡的設定 ``` $ ip a s 4: ens35: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vm-bo state UP group default qlen 1000 link/ether 00:0c:29:23:59:d5 brd ff:ff:ff:ff:ff:ff altname enp2s3 5: ens36: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vm-bo state UP group default qlen 1000 link/ether 00:0c:29:23:59:df brd ff:ff:ff:ff:ff:ff altname enp2s4 ... 65: vm-bo: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master vm-br state UP group default qlen 1000 link/ether 00:0c:29:23:59:d5 brd ff:ff:ff:ff:ff:ff 66: vm-br: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default link/ether 1e:df:20:6d:88:47 brd ff:ff:ff:ff:ff:ff ``` > 發現多了 `vm-bo` 這張網卡和 `vm-br` 這個 bridge 裝置 > 但是 `vm-br` 橋接裝置沒有自己的 IP ### 2.2. 備份 Harvester Configuration 檔案 `/oem/90_custom.yaml` ``` $ sudo cp /oem/90_custom.yaml /oem/install/90_custom.yaml.$(date --iso-8601=minutes) ``` ### 2.3. 修改 `/oem/90_custom.yaml` 檔案，並且更新以下 Yaml 檔中路徑的值 `stages.initramfs[0].files` ``` $ vi /oem/90_custom.yaml ``` 內容如下 ``` name: Harvester Configuration stages: initramfs: - commands: ... files: # add the following context - path: /etc/wicked/scripts/setup_vm_bond.sh permissions: 493 owner: 0 group: 0 content: | #!/bin/sh ACTION=$1 INTERFACE=$2 case $ACTION in post-up) # inherit MAC address ip link set dev vm-br address $(ip -json link show dev $INTERFACE | jq -j '.[0]["address"]') # accept all vlan, PVID=1 by default bridge vlan add vid 2-4094 dev $INTERFACE ;; esac encoding: "" ownerstring: "" - path: /etc/sysconfig/network/ifcfg-vm-bo permissions: 384 owner: 0 group: 0 content: |+ STARTMODE='onboot' BONDING_MASTER='yes' BOOTPROTO='none' POST_UP_SCRIPT="wicked:setup_vm_bond.sh" BONDING_SLAVE_0='ens35' BONDING_SLAVE_1='ens36' BONDING_MODULE_OPTS='miimon=100 mode=balance-alb ' DHCLIENT_SET_DEFAULT_ROUTE='no' encoding: "" - path: /etc/sysconfig/network/ifcfg-ens35 permissions: 384 owner: 0 group: 0 content: | STARTMODE='hotplug' BOOTPROTO='none' encoding: "" ownerstring: "" - path: /etc/sysconfig/network/ifcfg-ens36 permissions: 384 owner: 0 group: 0 content: | STARTMODE='hotplug' BOOTPROTO='none' encoding: "" ownerstring: "" - path: /etc/wicked/scripts/setup_vm_bridge.sh permissions: 493 owner: 0 group: 0 content: | #!/bin/sh ACTION=$1 INTERFACE=$2 case $ACTION in pre-up) # enable vlan-aware ip link set $INTERFACE type bridge vlan_filtering 1 ;; post-up) # accept all vlan, PVID=1 by default bridge vlan add vid 2-4094 dev $INTERFACE self bridge vlan add vid 2-4094 dev vm-bo ;; esac encoding: "" ownerstring: "" - path: /etc/sysconfig/network/ifcfg-vm-br permissions: 384 owner: 0 group: 0 content: |+ STARTMODE='onboot' BOOTPROTO='static' BRIDGE='yes' BRIDGE_STP='off' BRIDGE_FORWARDDELAY='0' BRIDGE_PORTS='vm-bo' PRE_UP_SCRIPT="wicked:setup_vm_bridge.sh" POST_UP_SCRIPT="wicked:setup_vm_bridge.sh" IPADDR=192.168.81.21 NETMASK=255.255.255.0 DHCLIENT_SET_DEFAULT_ROUTE='yes' encoding: "" ownerstring: "" ``` 1. `/etc/wicked/scripts/setup_vm_bond.sh` 修改內容如下 : - `ip link set dev <你的 bridge 名稱> address` - bridge 的名稱會是 `<cluster network name>-br` - 譬如: 我的 `cluster network` 取名為 `vm`，那麼名稱就是 `vm-br` - 此檔名可自訂義 2. `/etc/sysconfig/network/ifcfg-vm-bo` 修改內容如下 : - `POST_UP_SCRIPT="wicked:setup_vm_bond.sh"` - 格式 : `wicked:<第一個 script 的名稱>` - `BONDING_SLAVE_0='ens35'` - 格式 : `ONDING_SLAVE_0='第一張網卡名稱'` - `BONDING_SLAVE_1='ens36'` - 格式 : `ONDING_SLAVE_1='第二張網卡名稱'` - `BONDING_MODULE_OPTS='miimon=100 mode=balance-alb '` - 格式 : `BONDING_MODULE_OPTS='miimon=<在 UI 上建立的值> node=<在 UI 上選的 Bond mode>'` 3. `/etc/sysconfig/network/ifcfg-ens35` 修改內容如下 : - 檔名須為: `ifcfg-<網卡名稱>` 4. `/etc/sysconfig/network/ifcfg-ens36` 修改內容如下 : - 檔名須為: `ifcfg-<網卡名稱>` 5. `/etc/wicked/scripts/setup_vm_bridge.sh` 修改內容如下 : - 檔名可自訂義 6. `/etc/sysconfig/network/ifcfg-vm-br` 修改內容如下 : - `BRIDGE_PORTS='vm-bo'` - 格式: `BRIDGE_PORTS='bridge 名稱'` - `PRE_UP_SCRIPT="wicked:setup_vm_bridge.sh"` - 格式: `PRE_UP_SCRIPT="上一個 Script 檔案的名稱"` - `POST_UP_SCRIPT="wicked:setup_vm_bridge.sh"` - 格式: `POST_UP_SCRIPT="上一個 Script 檔案的名稱"` - `IPADDR=192.168.81.21` - 改成符合這張網卡的 IP Address - `NETMASK=255.255.255.0` - 改成符合這張網卡的子網路遮罩 ### 2.4. 重開機 ``` $ sudo init 6 ``` > 如果這台 node 上有正在跑的 VM 要先關機。 ### 2.5. 確認成功幫 Bridge 設靜態 IP ``` $ ip a s ... 4: ens35: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vm-bo state UP group default qlen 1000 link/ether 00:0c:29:f4:48:52 brd ff:ff:ff:ff:ff:ff altname enp2s3 5: ens36: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vm-bo state UP group default qlen 1000 link/ether 00:0c:29:f4:48:5c brd ff:ff:ff:ff:ff:ff altname enp2s4 6: vm-br: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 00:0c:29:f4:48:52 brd ff:ff:ff:ff:ff:ff inet 192.168.81.23/24 brd 192.168.81.255 scope global vm-br valid_lft forever preferred_lft forever 7: vm-bo: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master vm-br state UP group default qlen 1000 link/ether 00:0c:29:f4:48:52 brd ff:ff:ff:ff:ff:ff ``` ### 2.6. 重複以上 2.1. ~ 2.5. 的步驟依序做完每一台有在 `Cluster Netwrok` 底下的節點 這樣設定完 Harvester node 就能夠連到 VM ## 3. 建立 VM Network 點選左側選單 Networks -> VM Networks -> Create ![image](https://hackmd.io/_uploads/Syki2Fhi0.png) * **Name**: `vm` * **Type**: `UntaggedNetwrok` * **Cluster Network**: `vm` ![image](https://hackmd.io/_uploads/rk0l3YhiA.png) 建立成功，之前 VM 在建立的時候就能有自己獨立的網卡辣~ ![image](https://hackmd.io/_uploads/BJVP2YnsC.png)