# RKE2 切换 Calico IPIP & VXLAN 模式
## 環境準備
* 已安裝了 3 個節點 RKE2 叢集(3m)
* 所有節點都在同一個網段，驗證 VXLAN->IPIP->VXLAN 切換
> 節點 1: 192.168.11.102/24
> 節點 2: 192.168.11.152/24
> 節點 3: 192.168.11.153/24
```
$ kubectl get no -owide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE
KERNEL-VERSION CONTAINER-RUNTIME
rke2 Ready control-plane,etcd,master,worker 18h v1.27.8+rke2r1 192.168.11.102 <none> SUSE Linux Enterprise Server 15 SP6 6.4.0-150600.21-default containerd://1.7.7-k3s1
rke2-m2 Ready control-plane,etcd,master,worker 91m v1.27.8+rke2r1 192.168.11.152 <none> SUSE Linux Enterprise Server 15 SP6 6.4.0-150600.21-default containerd://1.7.7-k3s1
rke2-m3 Ready control-plane,etcd,master,worker 91m v1.27.8+rke2r1 192.168.11.153 <none> SUSE Linux Enterprise Server 15 SP6 6.4.0-150600.21-default containerd://1.7.7-k3s1
```
## 實作
### 切換 IPIP 模式
* 預設安裝的 calico 是 VXLAN 模式，驗證轉換為 IPIP
* 將以下 yaml 直接匯入 RKE2 叢集
```
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
name: rke2-calico
namespace: kube-system
spec:
valuesContent: |-
installation:
calicoNetwork:
bgp: Enabled
ipPools:
- cidr: 10.42.0.0/16 # 注意自己的 pod CIDR
encapsulation: IPIP
nodeAddressAutodetectionV4:
interface: eth0 # 注意自己的網卡名稱
```
* 修改 ippools
```
$ kubectl get ippools.crd.projectcalico.org/default-ipv4-ippool
NAME AGE
default-ipv4-ippool 18h
$ kubectl edit ippools.crd.projectcalico.org/default-ipv4-ippool
......
cidr: 10.42.0.0/16
ipipMode: Always # 修改此行
natOutgoing: true
nodeSelector: all()
vxlanMode: Never # 修改此行
```
* 儲存後，觀察 `calico-system/calico-node-XXXX` pod 會逐一重啟，並在節點上刪除 vxlan.calico 網卡，然後建立 tunl0 網卡。
* 如果發現有 pod 沒有重啟，就手動重啟所有 `calico-node-XXXX` pod。
```
$ kubectl -n calico-system get po
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-7bb5fb9c48-k7l4s 1/1 Running 0 18h
calico-node-8xnj6 1/1 Running 0 93s
calico-node-mt6nd 1/1 Running 0 93s
calico-node-nvrxs 1/1 Running 0 4m33s
calico-typha-544749c68f-27mdj 1/1 Running 0 18h
calico-typha-544749c68f-x4qkf 1/1 Running 0 107m
# 每個節點都會多這張網卡
$ ip a s tunl0
37: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN group default qlen 1000
link/ipip 0.0.0.0 brd 0.0.0.0
inet 10.42.149.92/32 scope global tunl0
valid_lft forever preferred_lft forever
```
* 檢視 IP 分配與路由規則
```
$ kubectl get ipamblocks.crd.projectcalico.org -o yaml | egrep 'affinity:|cidr'
affinity: host:rke2
cidr: 10.42.149.64/26
affinity: host:rke2-m3
cidr: 10.42.255.192/26
affinity: host:rke2-m2
cidr: 10.42.62.128/26
# pod 跨節點溝通都會透過 tunl0 這張網卡
$ ip r s
10.42.62.128/26 via 192.168.11.152 dev tunl0 proto bird
10.42.255.192/26 via 192.168.11.153 dev tunl0 proto bird onlink
.....
```
* 驗證在每個節點都創建 pod，並且都能夠互相溝通
```
$ kubectl create deploy test --image=nginx --replicas=3
$ kubectl get po -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
test-5746d4c59f-45hps 1/1 Running 0 20s 10.42.149.93 rke2 <none> <none>
test-5746d4c59f-4fb8l 1/1 Running 0 20s 10.42.62.133 rke2-m2 <none> <none>
test-5746d4c59f-r8l4l 1/1 Running 0 20s 10.42.255.196 rke2-m3 <none> <none>
$ kubectl exec test-5746d4c59f-45hps -- curl 10.42.62.133
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
$ kubectl exec test-5746d4c59f-45hps -- curl 10.42.255.196
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
$ kubectl delete deploy test
```
### 切換 VXLAN 模式
* 將以下 yaml 直接匯入 RKE2 叢集
```
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
name: rke2-calico
namespace: kube-system
spec:
valuesContent: |-
installation:
calicoNetwork:
bgp: Disabled
ipPools:
- cidr: 10.42.0.0/16 # 注意自己的 pod CIDR
encapsulation: VXLAN
nodeAddressAutodetectionV4:
interface: eth0 # 注意自己的網卡名稱
```
* 修改 ippool
```
$ kubectl edit ippools.crd.projectcalico.org/default-ipv4-ippool
......
cidr: 10.42.0.0/16
ipipMode: Never # 修改此行
natOutgoing: true
nodeSelector: all()
vxlanMode: Always # 修改此行
```
* 儲存後，在節點上可以看到新增的 vxlan.calico 網卡，但是 tunl0 網卡不會刪除。
```
$ ip a s vxlan.calico
45: vxlan.calico: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether 66:92:bf:18:e9:75 brd ff:ff:ff:ff:ff:ff
inet 10.42.149.94/32 scope global vxlan.calico
valid_lft forever preferred_lft forever
inet6 fe80::6492:bfff:fe18:e975/64 scope link
valid_lft forever preferred_lft forever
```
* 檢視 IP 分配與路由規則
```
$ kubectl get ipamblocks.crd.projectcalico.org -o yaml | egrep 'affinity:|cidr'
affinity: host:rke2
cidr: 10.42.149.64/26
affinity: host:rke2-m3
cidr: 10.42.255.192/26
affinity: host:rke2-m2
cidr: 10.42.62.128/26
# pod 跨節點溝通都會透過 vxlan.calico 這張網卡
$ ip r s
10.42.62.128/26 via 10.42.62.134 dev vxlan.calico onlink
10.42.255.192/26 via 10.42.255.197 dev vxlan.calico onlink
......
```
* 驗證在每個節點都創建 pod，並且都能夠互相溝通
```
$ kubectl create deploy test --image=nginx --replicas=3
$ kubectl get po -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
test-5746d4c59f-dvqkt 1/1 Running 0 14s 10.42.62.135 rke2-m2 <none> <none>
test-5746d4c59f-rdltn 1/1 Running 0 14s 10.42.255.198 rke2-m3 <none> <none>
test-5746d4c59f-wv9ks 1/1 Running 0 14s 10.42.149.95 rke2 <none> <none>
$ kubectl exec test-5746d4c59f-dvqkt -- curl 10.42.255.198
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
$ kubectl exec test-5746d4c59f-dvqkt -- curl 10.42.149.95
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 615 100 615 0 0 69327 0 --:--:-- --:--:-- --:--:-- 76875
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
$ kubectl delete deploy test
```
## 跨多網段節點設定
* 如果一個叢集的節點是跨不同的網段，那麼就要更改 `encapsulation`
* 其餘設定方式皆與前面流程相同
* 切換 IPIP
```
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
name: rke2-calico
namespace: kube-system
spec:
valuesContent: |-
installation:
calicoNetwork:
bgp: Enabled
ipPools:
- cidr: 10.42.0.0/16 # 注意自己的 pod CIDR
encapsulation: IPIPCrossSubnet # 更改此行
nodeAddressAutodetectionV4:
interface: eth0 # 注意自己的網卡名稱
```
* 切換 VXLAN
```
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
name: rke2-calico
namespace: kube-system
spec:
valuesContent: |-
installation:
calicoNetwork:
bgp: Disabled
ipPools:
- cidr: 10.42.0.0/16 # 注意自己的 pod CIDR
encapsulation: VXLANCrossSubnet # 更改此行
nodeAddressAutodetectionV4:
interface: eth0 # 注意自己的網卡名稱
```
## 參考
https://documentation.suse.com/external-tree/en-us/cloudnative/rke2/latest/zh/networking/basic_network_options.html
https://docs.tigera.io/calico/3.27/reference/installation/api#operator.tigera.io/v1.Installation
Sign in with Wallet
Connect another wallet