---
# System prepended metadata

title: the K8S worker node join to the master node via wireguard
tags: [k8s, wireguard]

---

the K8S worker node join to the master node via wireguard
===

The note record **Raspberry pi** worker node join to the **Master** node via wireguard.

Environment
  ~ Server
    * Ubuntu 16.04
    * no-ip service
  ~ Client
    * Raspbian

# Install wireguard


## Client

**Step 1.** Install `wg` cli


ref.
 * [raspberrypiwireguard](https://github.com/adrianmihalko/raspberrypiwireguard)

**Step 2.** adding network interface `wg0`, attach `ip` and set-up `wg0`

```shell=
#!/bin/bash
INTERFACE=wg0
LOCAL_IP=10.56.0.101/24

echo "add interface $INTERFACE"
ip link add $INTERFACE type wireguard

echo "add ip $LOCAL_IP to $INTERFACE"
ip addr add $LOCAL_IP dev $INTERFACE
ip link set $INTERFACE up
```

**Step 3.** gen-key

```shell=
wg genkey | tee privatekey | wg pubkey > publickey

cat privatekey
cat publickey
```

**Step 4.** configure `client.conf`

```shell=
[Interface]
PrivateKey = <CLIENT_PRIVATE_KEY>
ListenPort = 21555
DNS = 8.8.8.8

[Peer]
PublicKey = <SERVER_PUBLIC_KEY>
PersistentKeepalive = 10
Endpoint = <DOMAIN>:51261
AllowedIPs = 0.0.0.0/0
```

**Step 5.** enable wireguard!

```shell=
sudo wg setconf wg0 client.conf
```

**Step 6.** adding routing rule for wireguard

```shell=
# check net ipv4 ip forward
sysctl -w net.ipv4.ip_forward=1

iptables -A FORWARD -i wg0 -j ACCEPT
iptables -A FORWARD -o wg0 -j ACCEPT
iptables -t nat -A POSTROUTING -o wg0 -j MASQUERADE
```

ref.
* [raspberrypiwireguard](https://github.com/adrianmihalko/raspberrypiwireguard)

## Server

**Step 1.** Install `wg` cli

```shell= 
add-apt-repository ppa:wireguard/wireguard
apt-get update
apt-get install wireguard
```

**Step 2.** adding network interface `wg0`, attach `ip` and set-up `wg0`

```shell=
#!/bin/bash
INTERFACE=wg0
LOCAL_IP=10.56.0.100/24

echo "add interface $INTERFACE"
ip link add $INTERFACE type wireguard

echo "add ip $LOCAL_IP to $INTERFACE"
ip addr add $LOCAL_IP dev $INTERFACE
ip link set $INTERFACE up
```

**Step 3.** gen-key

```shell=
wg genkey | tee privatekey | wg pubkey > publickey

cat privatekey
cat publickey
```

**Step 4.** configure `server.conf` need to `CLIENT_PUBLIC_KEY`

```shell=
[Interface]
PrivateKey = <SERVER_PRIVATE_KEY>
ListenPort = 51261

[Peer]
PublicKey = <CLIENT_PUBLIC_KEY>
PersistentKeepalive = 10
AllowedIPs = 10.56.0.101/32
```

**Step 5.** enable wireguard

```shell=
sudo wg setconf wg0 server.conf
```

**Step 6.** adding routing rule for wireguard

```shell=
# check net ipv4 ip forward
sysctl -w net.ipv4.ip_forward=1

iptables -A FORWARD -i wg0 -j ACCEPT
iptables -A FORWARD -o wg0 -j ACCEPT
iptables -t nat -A POSTROUTING -o wg0 -j MASQUERADE

# 從 ppp0 進來 從 wg0 出去的, 狀態又是連線中 就 ACCEPT
iptables -A FORWARD -o wg0 -i ppp0 -m state --state RELATED,ESTABLISHED -j ACCEPT

# iptables -t nat -A POSTROUTING -o wg0 -j MASQUERADE
# iptables -A FORWARD -o wg0 ! -i wg0 -j ACCEPT
# iptables -A FORWARD -i wg0 ! -o wg0 -m state --state RELATED,ESTABLISHED -j ACCEPT
```


ref
* [wireguard install](https://www.wireguard.com/install/)
* [wireguard quick start](https://www.wireguard.com/quickstart/)

# kubeadm init

## prepaer config

1. `ip route add default`
2. `iptables POSTROUTING` 到 wg0

3. delete default route, add default route 10.56.0.100, ping 不到 10.56.0.100



`ip route` http://linux.vbird.org/linux_server/0140networkcommand.php#route
https://anyisalin.github.io/2018/11/21/fast-flexible-nat-to-nat-vpn-wireguard/

---

Modify config

sudo vi /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

--node-ip=10.56.0.101

```shell=
sudo systemctl daemon-reload
sudo systemctl restart kubelet
```

```shell=
#!/bin/bash

INTERFACE=wg0

echo set iptable interface: $INTERFACE

iptables -A FORWARD -i $INTERFACE -j ACCEPT
iptables -A FORWARD -o $INTERFACE -j ACCEPT

iptables -t nat -A POSTROUTING -o ppp0 -j MASQUERADE

# 從 ppp0 進來 從 wg0 出去的, 狀態又是連線中 就 ACCEPT
iptables -A FORWARD -o wg0 -i ppp0 -m state --state RELATED,ESTABLISHED -j ACCEPT
```


# K8S
## kubeadm init

**Notice** must be adding 
  * `--apiserver-advertise-address` : it's wireguard gateway ip
  * `--apiserver-cert-extra-sans` : for ddns domain
  * `--pod-network-cidr` : for cni

```
sudo swapoff -a && sudo kubeadm init \
--pod-network-cidr=192.168.0.0/16 \
--apiserver-cert-extra-sans=sian-home.ddns.net \
--apiserver-advertise-address=10.56.0.100
```

## worker of kubelet(Respberry Pi)

setup node ip of `kubelet` because one enable via vpn ip

```shell=
sudo vi /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
```

```shell=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS --node-ip=10.56.0.100`
```

### join woker node to the master

```shell=
join command
```


# Troubleshoting

#### 1. setting default gateway


https://anyisalin.github.io/2018/11/21/fast-flexible-nat-to-nat-vpn-wireguard/
http://man.linuxde.net/iptables
https://www.lijiaocn.com/%E6%8A%80%E5%B7%A7/2018/06/15/debug-linux-network.html
https://www.ubuntu-tw.org/modules/newbb/viewtopic.php?post_id=253866

https://www.cnblogs.com/EasonJim/p/8424731.html

#### 3. 使用 `flannel` 時, 拿到ip之後沒辦法 通過 Pod IP, 互 ping.

`ip -d link show` 觀察時, 發現 `flannel` 開在 defualt 網卡 而不是 `wg0`, 後來通過改 yaml 檔, 會變成以下結果
```
153: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1370 qdisc noqueue state UNKNOWN mode DEFAULT group default
    link/ether 9e:45:5d:d1:49:f8 brd ff:ff:ff:ff:ff:ff promiscuity 0
    vxlan id 1 local 10.56.0.100 dev wg0 srcport 0 0 dstport 8472 nolearning ageing 300 udpcsum addrgenmode none
157: veth62ca6a41@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1370 qdisc noqueue master cni0 state UP mode DEFAULT group default
```

flannel yaml 的部分需要加上 args, `--iface` 來指定 interface `wg0` 　


```
  containers:
  - name: kube-flannel
    image: quay.io/coreos/flannel:v0.10.0-amd64
    command:
    - /opt/bin/flanneld
    args:
    - --ip-masq
    - --kube-subnet-mgr
    - --iface=enp0s8
```
https://stackoverflow.com/questions/47845739/configuring-flannel-to-use-a-non-default-interface-in-kubernetes

#### 4. `Cluster IP` 在 Pod 內無法互 ping

進 pod 使用 `nslookup my-nginx.default.svc.cluster.local` 有找到 coreDNS 的 `Cluster IP`, 但是不通.

routing 在從 pod 出來之後, `ip route list` 內 沒有對 cluster ip 作處理的, routing(`ip route get CLUSTER_IP`)會從預設的 gateway 走 預設網卡 不走 `flannel.1`

最後在 routing table 加上, 對 `CLUSTER_IP` 的規則.
```shell=
ip route add 10.96.0.0/16 dev flannel.1
```

因為 cluster ip 的range 很大 沒辦法直接用, cidr 一次指定, 因此在 `kubeadm init` 時, 加上 
`--service-cluster-ip-range` 為 `10.254.0.0/24`

然後
```shell=
ip route add 10.254.0.0/24 dev flannel.1
```
為佳

https://stackoverflow.com/questions/34639185/communication-failing-between-kubernetes-nodes-and-clusterip
https://jimmysong.io/posts/ip-and-service-discovry-in-kubernetes/
###### tags: `wireguard` `k8s`