# RKE2 Calico Multus Pod 多網卡設定
* Multus 是 Kubernetes 的一個 CNI 插件，允許 Pod 設定使用多個網卡介面。
## 實作
* 透過 rancher 建立 rke2 (1m2w)叢集，CNI 選擇 `multus,calico`
* 部屬完成
```
$ kubectl get no
NAME STATUS ROLES AGE VERSION
multus Ready control-plane,etcd,master,worker 53m v1.30.6+rke2r1
multus2 Ready worker 51m v1.30.6+rke2r1
multus3 Ready worker 83s v1.30.6+rke2r1
# 啟用 multus 功能，每個節點會多出 multus pod
$ kubectl -n kube-system get po -l app=rke2-multus
NAME READY STATUS RESTARTS AGE
rke2-multus-fjw6n 1/1 Running 3 (69m ago) 70m
rke2-multus-m2k25 1/1 Running 0 68m
rke2-multus-pr6l9 1/1 Running 2 (17m ago) 18m
```
## 單節點 pod 網路測試
* `"subnet": "10.10.0.0/16"` : 定義 pod 第二張網卡所會拿到的 ip 範圍。
* `"type": "bridge"` : 就是 linux 的虛擬橋接器，讓 pod 的第二張網卡都會接到這個虛擬橋接器，此功能只能讓 pod 在同台節點上溝通。
* `"bridge": "mynet0"` : 虛擬橋接器叫 `mynet0`。
```
$ cat <<EOF | kubectl create -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: bridge-conf
spec:
config: '{
"cniVersion": "0.3.1",
"name": "mynet",
"type": "bridge",
"bridge": "mynet0",
"ipam": {
"type": "host-local",
"subnet": "10.10.0.0/16",
"rangeStart": "10.10.1.20",
"rangeEnd": "10.10.3.50",
"routes": [
{ "dst": "0.0.0.0/0" }
]
}
}'
EOF
$ kubectl get network-attachment-definitions
NAME AGE
bridge-conf 43s
# 在 pod 內使用 annotations 宣告要使用雙網卡，並在同節點上產生
$ cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
name: samplepod1
annotations:
k8s.v1.cni.cncf.io/networks: bridge-conf
spec:
containers:
- name: samplepod
image: taiwanese/debug.alp
tty: true
imagePullPolicy: IfNotPresent
nodeName: multus
---
apiVersion: v1
kind: Pod
metadata:
name: samplepod2
annotations:
k8s.v1.cni.cncf.io/networks: bridge-conf
spec:
containers:
- name: samplepod
image: taiwanese/debug.alp
tty: true
imagePullPolicy: IfNotPresent
nodeName: multus
EOF
```
* 檢查 pod 內部雙網卡
```
$ kubectl get po -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
samplepod1 1/1 Running 0 6s 10.42.125.98 multus <none> <none>
samplepod2 1/1 Running 0 6s 10.42.125.99 multus <none> <none>
$ kubectl exec samplepod1 -- ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0@if154: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP qlen 1000
link/ether d6:e7:38:d2:79:2e brd ff:ff:ff:ff:ff:ff
inet 10.42.125.98/32 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::d4e7:38ff:fed2:792e/64 scope link
valid_lft forever preferred_lft forever
3: net1@if155: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
link/ether 0a:0b:8f:d7:44:3c brd ff:ff:ff:ff:ff:ff
inet 10.10.1.21/16 brd 10.10.255.255 scope global net1
valid_lft forever preferred_lft forever
inet6 fe80::80b:8fff:fed7:443c/64 scope link
valid_lft forever preferred_lft forever
$ kubectl exec samplepod2 -- ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0@if156: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP qlen 1000
link/ether 3a:e8:66:7c:fb:ec brd ff:ff:ff:ff:ff:ff
inet 10.42.125.99/32 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::38e8:66ff:fe7c:fbec/64 scope link
valid_lft forever preferred_lft forever
3: net1@if157: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
link/ether 96:8f:96:57:7f:60 brd ff:ff:ff:ff:ff:ff
inet 10.10.1.22/16 brd 10.10.255.255 scope global net1
valid_lft forever preferred_lft forever
inet6 fe80::948f:96ff:fe57:7f60/64 scope link
valid_lft forever preferred_lft forever
```
* 驗證 pod 可以透過第二張網卡互通
```
$ kubectl exec samplepod1 -- ping -I net1 10.10.1.22
PING 10.10.1.22 (10.10.1.22): 56 data bytes
64 bytes from 10.10.1.22: seq=0 ttl=64 time=0.068 ms
64 bytes from 10.10.1.22: seq=1 ttl=64 time=0.051 ms
```
* 這兩個 pod 的第二張網卡都接到 `mynet0` 這個虛擬橋接器，因此可以互相溝通
```
$ brctl show
bridge name bridge id STP enabled interfaces
mynet0 8000.06e2415c59f1 no veth695dbfea
vethbdb47d75
```
* 環境清除
```
$ kubectl delete po samplepod1 samplepod2
$ kubectl delete network-attachment-definitions bridge-conf
```
## 跨節點 pod 網路測試
* 建立基於 macvlan 的額外網路。可讓節點上的 Pod 可以透過實體網卡與其他節點上的 Pod (包含節點本身)溝通。附加到建立基於 macvlan 的額外網路的每個 Pod 都會取得一個唯一的 MAC。
```
$ cat <<EOF | kubectl create -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: macvlan-conf
spec:
config: '{
"cniVersion": "0.3.1",
"name": "mynet",
"type": "macvlan",
"ipam": {
"type": "host-local",
"subnet": "10.10.0.0/16",
"rangeStart": "10.10.1.20",
"rangeEnd": "10.10.3.50",
"routes": [
{ "dst": "0.0.0.0/0" }
]
}
}'
EOF
$ kubectl get network-attachment-definitions
NAME AGE
macvlan-conf 29s
# 在 pod 內使用 annotations 宣告要使用雙網卡，並在不同節點上產生
$ cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
name: samplepod1
annotations:
k8s.v1.cni.cncf.io/networks: macvlan-conf
spec:
containers:
- name: samplepod
image: taiwanese/debug.alp
tty: true
imagePullPolicy: IfNotPresent
nodeName: multus
---
apiVersion: v1
kind: Pod
metadata:
name: samplepod2
annotations:
k8s.v1.cni.cncf.io/networks: macvlan-conf
spec:
containers:
- name: samplepod
image: taiwanese/debug.alp
tty: true
imagePullPolicy: IfNotPresent
nodeName: multus2
EOF
```
```
$ kubectl get po -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
samplepod1 1/1 Running 0 13s 10.42.125.109 multus <none> <none>
samplepod2 1/1 Running 0 13s 10.42.65.125 multus2 <none> <none>
$ kubectl exec samplepod1 -- ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0@if176: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP qlen 1000
link/ether 9e:a9:c4:73:0a:fc brd ff:ff:ff:ff:ff:ff
inet 10.42.125.109/32 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::9ca9:c4ff:fe73:afc/64 scope link
valid_lft forever preferred_lft forever
3: net1@net1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether 5e:cb:92:ad:30:19 brd ff:ff:ff:ff:ff:ff
inet 10.10.1.32/16 brd 10.10.255.255 scope global net1
valid_lft forever preferred_lft forever
inet6 fe80::5ccb:92ff:fead:3019/64 scope link
valid_lft forever preferred_lft forever
$ kubectl exec samplepod2 -- ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0@if130: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP qlen 1000
link/ether aa:ff:c6:1e:47:e7 brd ff:ff:ff:ff:ff:ff
inet 10.42.65.125/32 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::a8ff:c6ff:fe1e:47e7/64 scope link
valid_lft forever preferred_lft forever
3: net1@net1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether 6a:df:2d:2c:bf:04 brd ff:ff:ff:ff:ff:ff
inet 10.10.1.22/16 brd 10.10.255.255 scope global net1
valid_lft forever preferred_lft forever
inet6 fe80::68df:2dff:fe2c:bf04/64 scope link
valid_lft forever preferred_lft forever
```
* 驗證 pod 之間可以透過第二張網卡跨節點溝通
```
$ kubectl exec samplepod1 -- ping -I net1 10.10.1.22
PING 10.10.1.22 (10.10.1.22): 56 data bytes
64 bytes from 10.10.1.22: seq=0 ttl=64 time=0.224 ms
64 bytes from 10.10.1.22: seq=1 ttl=64 time=0.095 ms
```
* 環境清除
```
$ kubectl delete po samplepod1 samplepod2
```
## 效能測試
```
$ echo 'apiVersion: apps/v1
kind: DaemonSet
metadata:
name: ds-iperf
namespace: default
spec:
selector:
matchLabels:
app: iperf3
template:
metadata:
annotations:
k8s.v1.cni.cncf.io/networks: macvlan-conf
labels:
app: iperf3
spec:
containers:
- name: iperf3
image: leodotcloud/swiss-army-knife
command: ["iperf3"]
args: ["-s", "-p 12345"]
ports:
- containerPort: 12345' | kubectl apply -f -
```
* 檢視 pod ip
```
$ kubectl get po -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
ds-iperf-8x9bd 1/1 Running 0 21m 10.42.65.70 multus2 <none> <none>
ds-iperf-dfkfk 1/1 Running 0 88s 10.42.102.66 multus3 <none> <none>
ds-iperf-lmdtk 1/1 Running 0 21m 10.42.125.95 multus <none> <none>
$ kubectl exec ds-iperf-8x9bd -- ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0@if11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
link/ether 6a:47:24:dc:93:eb brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.42.65.70/32 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::6847:24ff:fedc:93eb/64 scope link
valid_lft forever preferred_lft forever
3: net1@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 9e:75:86:97:e3:c5 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.10.1.22/16 brd 10.10.255.255 scope global net1
valid_lft forever preferred_lft forever
inet6 fe80::9c75:86ff:fe97:e3c5/64 scope link
valid_lft forever preferred_lft forever
$ kubectl exec ds-iperf-dfkfk -- ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
link/ether ce:91:7e:d0:52:83 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.42.102.66/32 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::cc91:7eff:fed0:5283/64 scope link
valid_lft forever preferred_lft forever
3: net1@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 46:fc:48:86:75:7e brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.10.1.20/16 brd 10.10.255.255 scope global net1
valid_lft forever preferred_lft forever
inet6 fe80::44fc:48ff:fe86:757e/64 scope link
valid_lft forever preferred_lft forever
```
* 測試 pod 跨節點且跨實體機網路效能。
* 使用 calico vxlan 網路壓測，網路頻寬為 `5.15 Gbits/sec`
```
$ kubectl exec ds-iperf-8x9bd -- iperf3 -c 10.42.102.66 -p 12345
Connecting to host 10.42.102.66, port 12345
[ 4] local 10.42.65.70 port 42994 connected to 10.42.102.66 port 12345
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 585 MBytes 4.90 Gbits/sec 8 1.54 MBytes
[ 4] 1.00-2.00 sec 591 MBytes 4.96 Gbits/sec 14 1.25 MBytes
[ 4] 2.00-3.00 sec 590 MBytes 4.95 Gbits/sec 608 1.51 MBytes
[ 4] 3.00-4.00 sec 599 MBytes 5.03 Gbits/sec 12 1.27 MBytes
[ 4] 4.00-5.00 sec 589 MBytes 4.94 Gbits/sec 0 1.56 MBytes
[ 4] 5.00-6.00 sec 635 MBytes 5.33 Gbits/sec 5 1.34 MBytes
[ 4] 6.00-7.00 sec 608 MBytes 5.09 Gbits/sec 0 1.63 MBytes
[ 4] 7.00-8.00 sec 645 MBytes 5.41 Gbits/sec 6 1.42 MBytes
[ 4] 8.00-9.00 sec 652 MBytes 5.47 Gbits/sec 0 1.71 MBytes
[ 4] 9.00-10.00 sec 645 MBytes 5.41 Gbits/sec 5 1.51 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 5.99 GBytes 5.15 Gbits/sec 658 sender
[ 4] 0.00-10.00 sec 5.99 GBytes 5.15 Gbits/sec receiver
iperf Done.
```
* 實體網路為 10g ，而使用第二張網卡 macvlan 網路壓測，網路頻寬為 `9.40 Gbits/sec` 幾乎可以用滿 10g 網路。
```
$ kubectl exec ds-iperf-8x9bd -- iperf3 -c 10.10.1.20 -p 12345
Connecting to host 10.10.1.20, port 12345
[ 4] local 10.10.1.22 port 37720 connected to 10.10.1.20 port 12345
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 1.10 GBytes 9.42 Gbits/sec 0 3.14 MBytes
[ 4] 1.00-2.00 sec 1.09 GBytes 9.39 Gbits/sec 0 3.14 MBytes
[ 4] 2.00-3.00 sec 1.09 GBytes 9.37 Gbits/sec 0 3.14 MBytes
[ 4] 3.00-4.00 sec 1.09 GBytes 9.40 Gbits/sec 0 3.14 MBytes
[ 4] 4.00-5.00 sec 1.09 GBytes 9.41 Gbits/sec 0 3.14 MBytes
[ 4] 5.00-6.00 sec 1.09 GBytes 9.40 Gbits/sec 0 3.14 MBytes
[ 4] 6.00-7.00 sec 1.10 GBytes 9.41 Gbits/sec 0 3.14 MBytes
[ 4] 7.00-8.00 sec 1.09 GBytes 9.40 Gbits/sec 0 3.14 MBytes
[ 4] 8.00-9.00 sec 1.09 GBytes 9.40 Gbits/sec 0 3.14 MBytes
[ 4] 9.00-10.00 sec 1.09 GBytes 9.40 Gbits/sec 0 3.14 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 10.9 GBytes 9.40 Gbits/sec 0 sender
[ 4] 0.00-10.00 sec 10.9 GBytes 9.40 Gbits/sec receiver
iperf Done.
```
### 結論: container 透過多網卡讓網路分流可以有效提升效能
## 參考
https://github.com/k8snetworkplumbingwg/multus-cni/blob/master/docs/how-to-use.md
Sign in with Wallet
Connect another wallet