---
# System prepended metadata

title: RKE2 Calico Multus Pod 多網卡設定
tags: [work]

---

# RKE2 Calico Multus Pod 多網卡設定
* Multus 是 Kubernetes 的一個 CNI 插件，允許 Pod 設定使用多個網卡介面。


## 實作
* 透過 rancher 建立 rke2 (1m2w)叢集，CNI 選擇 `multus,calico`

![image](https://hackmd.io/_uploads/ByfybaqMye.png)


* 部屬完成
```
$ kubectl get no
NAME      STATUS   ROLES                              AGE   VERSION
multus    Ready    control-plane,etcd,master,worker   53m   v1.30.6+rke2r1
multus2   Ready    worker                             51m   v1.30.6+rke2r1
multus3   Ready    worker                             83s   v1.30.6+rke2r1

# 啟用 multus 功能，每個節點會多出 multus pod
$ kubectl -n kube-system get po -l app=rke2-multus
NAME                READY   STATUS    RESTARTS      AGE
rke2-multus-fjw6n   1/1     Running   3 (69m ago)   70m
rke2-multus-m2k25   1/1     Running   0             68m
rke2-multus-pr6l9   1/1     Running   2 (17m ago)   18m
```

## 單節點 pod 網路測試
* `"subnet": "10.10.0.0/16"` : 定義 pod 第二張網卡所會拿到的 ip 範圍。
* `"type": "bridge"` : 就是 linux 的虛擬橋接器，讓 pod 的第二張網卡都會接到這個虛擬橋接器，此功能只能讓 pod 在同台節點上溝通。
* `"bridge": "mynet0"` : 虛擬橋接器叫 `mynet0`。
```
$ cat <<EOF | kubectl create -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: bridge-conf
spec:
  config: '{
            "cniVersion": "0.3.1",
            "name": "mynet",
            "type": "bridge",
            "bridge": "mynet0",
            "ipam": {
              "type": "host-local",
              "subnet": "10.10.0.0/16",
              "rangeStart": "10.10.1.20",
              "rangeEnd": "10.10.3.50",
              "routes": [
                { "dst": "0.0.0.0/0" }
              ]
            }
          }'
EOF

$ kubectl get network-attachment-definitions
NAME          AGE
bridge-conf   43s

# 在 pod 內使用 annotations 宣告要使用雙網卡，並在同節點上產生
$ cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
  name: samplepod1
  annotations:
    k8s.v1.cni.cncf.io/networks: bridge-conf
spec:
  containers:
  - name: samplepod
    image: taiwanese/debug.alp
    tty: true
    imagePullPolicy: IfNotPresent
  nodeName: multus
---
apiVersion: v1
kind: Pod
metadata:
  name: samplepod2
  annotations:
    k8s.v1.cni.cncf.io/networks: bridge-conf
spec:
  containers:
  - name: samplepod
    image: taiwanese/debug.alp
    tty: true
    imagePullPolicy: IfNotPresent
  nodeName: multus
EOF
```
* 檢查 pod 內部雙網卡
```
$ kubectl get po -owide
NAME         READY   STATUS    RESTARTS   AGE   IP             NODE     NOMINATED NODE   READINESS GATES
samplepod1   1/1     Running   0          6s    10.42.125.98   multus   <none>           <none>
samplepod2   1/1     Running   0          6s    10.42.125.99   multus   <none>           <none>

$ kubectl exec samplepod1 -- ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0@if154: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP qlen 1000
    link/ether d6:e7:38:d2:79:2e brd ff:ff:ff:ff:ff:ff
    inet 10.42.125.98/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::d4e7:38ff:fed2:792e/64 scope link
       valid_lft forever preferred_lft forever
3: net1@if155: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
    link/ether 0a:0b:8f:d7:44:3c brd ff:ff:ff:ff:ff:ff
    inet 10.10.1.21/16 brd 10.10.255.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::80b:8fff:fed7:443c/64 scope link
       valid_lft forever preferred_lft forever
       
$ kubectl exec samplepod2 -- ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0@if156: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP qlen 1000
    link/ether 3a:e8:66:7c:fb:ec brd ff:ff:ff:ff:ff:ff
    inet 10.42.125.99/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::38e8:66ff:fe7c:fbec/64 scope link
       valid_lft forever preferred_lft forever
3: net1@if157: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
    link/ether 96:8f:96:57:7f:60 brd ff:ff:ff:ff:ff:ff
    inet 10.10.1.22/16 brd 10.10.255.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::948f:96ff:fe57:7f60/64 scope link
       valid_lft forever preferred_lft forever   
```


* 驗證 pod 可以透過第二張網卡互通
```
$ kubectl exec samplepod1 -- ping -I net1 10.10.1.22
PING 10.10.1.22 (10.10.1.22): 56 data bytes
64 bytes from 10.10.1.22: seq=0 ttl=64 time=0.068 ms
64 bytes from 10.10.1.22: seq=1 ttl=64 time=0.051 ms
```

* 這兩個 pod 的第二張網卡都接到 `mynet0` 這個虛擬橋接器，因此可以互相溝通
```
$ brctl show
bridge name     bridge id               STP enabled     interfaces
mynet0          8000.06e2415c59f1       no              veth695dbfea
                                                        vethbdb47d75
```
* 環境清除
```
$ kubectl delete po samplepod1 samplepod2

$ kubectl delete network-attachment-definitions bridge-conf
```
## 跨節點 pod 網路測試
* 建立基於 macvlan 的額外網路。可讓節點上的 Pod 可以透過實體網卡與其他節點上的 Pod (包含節點本身)溝通。附加到建立基於 macvlan 的額外網路的每個 Pod 都會取得一個唯一的 MAC。


```
$ cat <<EOF | kubectl create -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: macvlan-conf
spec:
  config: '{
            "cniVersion": "0.3.1",
            "name": "mynet",
            "type": "macvlan",
            "ipam": {
              "type": "host-local",
              "subnet": "10.10.0.0/16",
              "rangeStart": "10.10.1.20",
              "rangeEnd": "10.10.3.50",
              "routes": [
                { "dst": "0.0.0.0/0" }
              ]
            }
          }'
EOF

$ kubectl get network-attachment-definitions
NAME           AGE
macvlan-conf   29s

# 在 pod 內使用 annotations 宣告要使用雙網卡，並在不同節點上產生
$ cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
  name: samplepod1
  annotations:
    k8s.v1.cni.cncf.io/networks: macvlan-conf
spec:
  containers:
  - name: samplepod
    image: taiwanese/debug.alp
    tty: true
    imagePullPolicy: IfNotPresent
  nodeName: multus
---
apiVersion: v1
kind: Pod
metadata:
  name: samplepod2
  annotations:
    k8s.v1.cni.cncf.io/networks: macvlan-conf
spec:
  containers:
  - name: samplepod
    image: taiwanese/debug.alp
    tty: true
    imagePullPolicy: IfNotPresent
  nodeName: multus2
EOF
```
```
$ kubectl get po -owide
NAME         READY   STATUS    RESTARTS   AGE   IP              NODE      NOMINATED NODE   READINESS GATES
samplepod1   1/1     Running   0          13s   10.42.125.109   multus    <none>           <none>
samplepod2   1/1     Running   0          13s   10.42.65.125    multus2   <none>           <none>

$ kubectl exec samplepod1 -- ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0@if176: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP qlen 1000
    link/ether 9e:a9:c4:73:0a:fc brd ff:ff:ff:ff:ff:ff
    inet 10.42.125.109/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::9ca9:c4ff:fe73:afc/64 scope link
       valid_lft forever preferred_lft forever
3: net1@net1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether 5e:cb:92:ad:30:19 brd ff:ff:ff:ff:ff:ff
    inet 10.10.1.32/16 brd 10.10.255.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::5ccb:92ff:fead:3019/64 scope link
       valid_lft forever preferred_lft forever

$ kubectl exec samplepod2 -- ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0@if130: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP qlen 1000
    link/ether aa:ff:c6:1e:47:e7 brd ff:ff:ff:ff:ff:ff
    inet 10.42.65.125/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::a8ff:c6ff:fe1e:47e7/64 scope link
       valid_lft forever preferred_lft forever
3: net1@net1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether 6a:df:2d:2c:bf:04 brd ff:ff:ff:ff:ff:ff
    inet 10.10.1.22/16 brd 10.10.255.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::68df:2dff:fe2c:bf04/64 scope link
       valid_lft forever preferred_lft forever
```
* 驗證 pod 之間可以透過第二張網卡跨節點溝通
```
$ kubectl exec samplepod1 -- ping -I net1 10.10.1.22
PING 10.10.1.22 (10.10.1.22): 56 data bytes
64 bytes from 10.10.1.22: seq=0 ttl=64 time=0.224 ms
64 bytes from 10.10.1.22: seq=1 ttl=64 time=0.095 ms
```

* 環境清除
```
$ kubectl delete po samplepod1 samplepod2
```

## 效能測試

```
$ echo 'apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: ds-iperf
  namespace: default
spec:
  selector:
    matchLabels:
      app: iperf3
  template:
    metadata:
      annotations:
        k8s.v1.cni.cncf.io/networks: macvlan-conf
      labels:
        app: iperf3
    spec:
      containers:
      - name: iperf3
        image: leodotcloud/swiss-army-knife
        command: ["iperf3"]
        args: ["-s", "-p 12345"]
        ports:
        - containerPort: 12345' | kubectl apply -f -
```
* 檢視 pod ip
```
$ kubectl get po -owide
NAME             READY   STATUS    RESTARTS   AGE   IP             NODE      NOMINATED NODE   READINESS GATES
ds-iperf-8x9bd   1/1     Running   0          21m   10.42.65.70    multus2   <none>           <none>
ds-iperf-dfkfk   1/1     Running   0          88s   10.42.102.66   multus3   <none>           <none>
ds-iperf-lmdtk   1/1     Running   0          21m   10.42.125.95   multus    <none>           <none>

$ kubectl exec ds-iperf-8x9bd -- ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0@if11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether 6a:47:24:dc:93:eb brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.42.65.70/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::6847:24ff:fedc:93eb/64 scope link
       valid_lft forever preferred_lft forever
3: net1@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 9e:75:86:97:e3:c5 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.10.1.22/16 brd 10.10.255.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::9c75:86ff:fe97:e3c5/64 scope link
       valid_lft forever preferred_lft forever
       
$ kubectl exec ds-iperf-dfkfk -- ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether ce:91:7e:d0:52:83 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.42.102.66/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::cc91:7eff:fed0:5283/64 scope link
       valid_lft forever preferred_lft forever
3: net1@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 46:fc:48:86:75:7e brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.10.1.20/16 brd 10.10.255.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::44fc:48ff:fe86:757e/64 scope link
       valid_lft forever preferred_lft forever
```
* 測試 pod 跨節點且跨實體機網路效能。
* 使用 calico vxlan 網路壓測，網路頻寬為 `5.15 Gbits/sec`

```
$ kubectl exec ds-iperf-8x9bd -- iperf3 -c 10.42.102.66 -p 12345
Connecting to host 10.42.102.66, port 12345
[  4] local 10.42.65.70 port 42994 connected to 10.42.102.66 port 12345
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   585 MBytes  4.90 Gbits/sec    8   1.54 MBytes
[  4]   1.00-2.00   sec   591 MBytes  4.96 Gbits/sec   14   1.25 MBytes
[  4]   2.00-3.00   sec   590 MBytes  4.95 Gbits/sec  608   1.51 MBytes
[  4]   3.00-4.00   sec   599 MBytes  5.03 Gbits/sec   12   1.27 MBytes
[  4]   4.00-5.00   sec   589 MBytes  4.94 Gbits/sec    0   1.56 MBytes
[  4]   5.00-6.00   sec   635 MBytes  5.33 Gbits/sec    5   1.34 MBytes
[  4]   6.00-7.00   sec   608 MBytes  5.09 Gbits/sec    0   1.63 MBytes
[  4]   7.00-8.00   sec   645 MBytes  5.41 Gbits/sec    6   1.42 MBytes
[  4]   8.00-9.00   sec   652 MBytes  5.47 Gbits/sec    0   1.71 MBytes
[  4]   9.00-10.00  sec   645 MBytes  5.41 Gbits/sec    5   1.51 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  5.99 GBytes  5.15 Gbits/sec  658             sender
[  4]   0.00-10.00  sec  5.99 GBytes  5.15 Gbits/sec                  receiver

iperf Done.
```
![image](https://hackmd.io/_uploads/Bk6soXsM1l.png)


* 實體網路為 10g ，而使用第二張網卡 macvlan 網路壓測，網路頻寬為 `9.40 Gbits/sec` 幾乎可以用滿 10g 網路。
```
$ kubectl exec ds-iperf-8x9bd -- iperf3 -c 10.10.1.20 -p 12345
Connecting to host 10.10.1.20, port 12345
[  4] local 10.10.1.22 port 37720 connected to 10.10.1.20 port 12345
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  1.10 GBytes  9.42 Gbits/sec    0   3.14 MBytes
[  4]   1.00-2.00   sec  1.09 GBytes  9.39 Gbits/sec    0   3.14 MBytes
[  4]   2.00-3.00   sec  1.09 GBytes  9.37 Gbits/sec    0   3.14 MBytes
[  4]   3.00-4.00   sec  1.09 GBytes  9.40 Gbits/sec    0   3.14 MBytes
[  4]   4.00-5.00   sec  1.09 GBytes  9.41 Gbits/sec    0   3.14 MBytes
[  4]   5.00-6.00   sec  1.09 GBytes  9.40 Gbits/sec    0   3.14 MBytes
[  4]   6.00-7.00   sec  1.10 GBytes  9.41 Gbits/sec    0   3.14 MBytes
[  4]   7.00-8.00   sec  1.09 GBytes  9.40 Gbits/sec    0   3.14 MBytes
[  4]   8.00-9.00   sec  1.09 GBytes  9.40 Gbits/sec    0   3.14 MBytes
[  4]   9.00-10.00  sec  1.09 GBytes  9.40 Gbits/sec    0   3.14 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  10.9 GBytes  9.40 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  10.9 GBytes  9.40 Gbits/sec                  receiver

iperf Done.
```
![image](https://hackmd.io/_uploads/HJA2jXsfke.png)

### 結論: container 透過多網卡讓網路分流可以有效提升效能

## 參考
https://github.com/k8snetworkplumbingwg/multus-cni/blob/master/docs/how-to-use.md