the K8S worker node join to the master node via wireguard === The note record **Raspberry pi** worker node join to the **Master** node via wireguard. Environment ~ Server * Ubuntu 16.04 * no-ip service ~ Client * Raspbian # Install wireguard ## Client **Step 1.** Install `wg` cli ref. * [raspberrypiwireguard](https://github.com/adrianmihalko/raspberrypiwireguard) **Step 2.** adding network interface `wg0`, attach `ip` and set-up `wg0` ```shell= #!/bin/bash INTERFACE=wg0 LOCAL_IP=10.56.0.101/24 echo "add interface $INTERFACE" ip link add $INTERFACE type wireguard echo "add ip $LOCAL_IP to $INTERFACE" ip addr add $LOCAL_IP dev $INTERFACE ip link set $INTERFACE up ``` **Step 3.** gen-key ```shell= wg genkey | tee privatekey | wg pubkey > publickey cat privatekey cat publickey ``` **Step 4.** configure `client.conf` ```shell= [Interface] PrivateKey = <CLIENT_PRIVATE_KEY> ListenPort = 21555 DNS = 8.8.8.8 [Peer] PublicKey = <SERVER_PUBLIC_KEY> PersistentKeepalive = 10 Endpoint = <DOMAIN>:51261 AllowedIPs = 0.0.0.0/0 ``` **Step 5.** enable wireguard! ```shell= sudo wg setconf wg0 client.conf ``` **Step 6.** adding routing rule for wireguard ```shell= # check net ipv4 ip forward sysctl -w net.ipv4.ip_forward=1 iptables -A FORWARD -i wg0 -j ACCEPT iptables -A FORWARD -o wg0 -j ACCEPT iptables -t nat -A POSTROUTING -o wg0 -j MASQUERADE ``` ref. * [raspberrypiwireguard](https://github.com/adrianmihalko/raspberrypiwireguard) ## Server **Step 1.** Install `wg` cli ```shell= add-apt-repository ppa:wireguard/wireguard apt-get update apt-get install wireguard ``` **Step 2.** adding network interface `wg0`, attach `ip` and set-up `wg0` ```shell= #!/bin/bash INTERFACE=wg0 LOCAL_IP=10.56.0.100/24 echo "add interface $INTERFACE" ip link add $INTERFACE type wireguard echo "add ip $LOCAL_IP to $INTERFACE" ip addr add $LOCAL_IP dev $INTERFACE ip link set $INTERFACE up ``` **Step 3.** gen-key ```shell= wg genkey | tee privatekey | wg pubkey > publickey cat privatekey cat publickey ``` **Step 4.** configure `server.conf` need to `CLIENT_PUBLIC_KEY` ```shell= [Interface] PrivateKey = <SERVER_PRIVATE_KEY> ListenPort = 51261 [Peer] PublicKey = <CLIENT_PUBLIC_KEY> PersistentKeepalive = 10 AllowedIPs = 10.56.0.101/32 ``` **Step 5.** enable wireguard ```shell= sudo wg setconf wg0 server.conf ``` **Step 6.** adding routing rule for wireguard ```shell= # check net ipv4 ip forward sysctl -w net.ipv4.ip_forward=1 iptables -A FORWARD -i wg0 -j ACCEPT iptables -A FORWARD -o wg0 -j ACCEPT iptables -t nat -A POSTROUTING -o wg0 -j MASQUERADE # 從 ppp0 進來 從 wg0 出去的, 狀態又是連線中 就 ACCEPT iptables -A FORWARD -o wg0 -i ppp0 -m state --state RELATED,ESTABLISHED -j ACCEPT # iptables -t nat -A POSTROUTING -o wg0 -j MASQUERADE # iptables -A FORWARD -o wg0 ! -i wg0 -j ACCEPT # iptables -A FORWARD -i wg0 ! -o wg0 -m state --state RELATED,ESTABLISHED -j ACCEPT ``` ref * [wireguard install](https://www.wireguard.com/install/) * [wireguard quick start](https://www.wireguard.com/quickstart/) # kubeadm init ## prepaer config 1. `ip route add default` 2. `iptables POSTROUTING` 到 wg0 3. delete default route, add default route 10.56.0.100, ping 不到 10.56.0.100 `ip route` http://linux.vbird.org/linux_server/0140networkcommand.php#route https://anyisalin.github.io/2018/11/21/fast-flexible-nat-to-nat-vpn-wireguard/ --- Modify config sudo vi /etc/systemd/system/kubelet.service.d/10-kubeadm.conf --node-ip=10.56.0.101 ```shell= sudo systemctl daemon-reload sudo systemctl restart kubelet ``` ```shell= #!/bin/bash INTERFACE=wg0 echo set iptable interface: $INTERFACE iptables -A FORWARD -i $INTERFACE -j ACCEPT iptables -A FORWARD -o $INTERFACE -j ACCEPT iptables -t nat -A POSTROUTING -o ppp0 -j MASQUERADE # 從 ppp0 進來 從 wg0 出去的, 狀態又是連線中 就 ACCEPT iptables -A FORWARD -o wg0 -i ppp0 -m state --state RELATED,ESTABLISHED -j ACCEPT ``` # K8S ## kubeadm init **Notice** must be adding * `--apiserver-advertise-address` : it's wireguard gateway ip * `--apiserver-cert-extra-sans` : for ddns domain * `--pod-network-cidr` : for cni ``` sudo swapoff -a && sudo kubeadm init \ --pod-network-cidr=192.168.0.0/16 \ --apiserver-cert-extra-sans=sian-home.ddns.net \ --apiserver-advertise-address=10.56.0.100 ``` ## worker of kubelet(Respberry Pi) setup node ip of `kubelet` because one enable via vpn ip ```shell= sudo vi /etc/systemd/system/kubelet.service.d/10-kubeadm.conf ``` ```shell= ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS --node-ip=10.56.0.100` ``` ### join woker node to the master ```shell= join command ``` # Troubleshoting #### 1. setting default gateway https://anyisalin.github.io/2018/11/21/fast-flexible-nat-to-nat-vpn-wireguard/ http://man.linuxde.net/iptables https://www.lijiaocn.com/%E6%8A%80%E5%B7%A7/2018/06/15/debug-linux-network.html https://www.ubuntu-tw.org/modules/newbb/viewtopic.php?post_id=253866 https://www.cnblogs.com/EasonJim/p/8424731.html #### 3. 使用 `flannel` 時, 拿到ip之後沒辦法 通過 Pod IP, 互 ping. `ip -d link show` 觀察時, 發現 `flannel` 開在 defualt 網卡 而不是 `wg0`, 後來通過改 yaml 檔, 會變成以下結果 ``` 153: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1370 qdisc noqueue state UNKNOWN mode DEFAULT group default link/ether 9e:45:5d:d1:49:f8 brd ff:ff:ff:ff:ff:ff promiscuity 0 vxlan id 1 local 10.56.0.100 dev wg0 srcport 0 0 dstport 8472 nolearning ageing 300 udpcsum addrgenmode none 157: veth62ca6a41@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1370 qdisc noqueue master cni0 state UP mode DEFAULT group default ``` flannel yaml 的部分需要加上 args, `--iface` 來指定 interface `wg0` 　 ``` containers: - name: kube-flannel image: quay.io/coreos/flannel:v0.10.0-amd64 command: - /opt/bin/flanneld args: - --ip-masq - --kube-subnet-mgr - --iface=enp0s8 ``` https://stackoverflow.com/questions/47845739/configuring-flannel-to-use-a-non-default-interface-in-kubernetes #### 4. `Cluster IP` 在 Pod 內無法互 ping 進 pod 使用 `nslookup my-nginx.default.svc.cluster.local` 有找到 coreDNS 的 `Cluster IP`, 但是不通. routing 在從 pod 出來之後, `ip route list` 內 沒有對 cluster ip 作處理的, routing(`ip route get CLUSTER_IP`)會從預設的 gateway 走 預設網卡 不走 `flannel.1` 最後在 routing table 加上, 對 `CLUSTER_IP` 的規則. ```shell= ip route add 10.96.0.0/16 dev flannel.1 ``` 因為 cluster ip 的range 很大 沒辦法直接用, cidr 一次指定, 因此在 `kubeadm init` 時, 加上 `--service-cluster-ip-range` 為 `10.254.0.0/24` 然後 ```shell= ip route add 10.254.0.0/24 dev flannel.1 ``` 為佳 https://stackoverflow.com/questions/34639185/communication-failing-between-kubernetes-nodes-and-clusterip https://jimmysong.io/posts/ip-and-service-discovry-in-kubernetes/ ###### tags: `wireguard` `k8s`