---
# System prepended metadata

title: 1. 虛擬機安裝kubernetes之方法
tags: [虛擬機流程]

---

# 1. 虛擬機安裝kubernetes之方法

###### tags: `虛擬機流程` 
## 第一步：環境搭建 

**一共3台master(control plane) 4台worker**

使用Virtual Box建立虛擬機
可到此[下載](https://www.virtualbox.org/wiki/Downloads)
下載Ubuntu server[印象檔](http://ftp.ubuntu-tw.org/ubuntu-releases/20.04/)

需要將CPU開啟虛擬化(進Bios開啟VT)才能使用64位元的映像檔

新增虛擬機，共7台

![](https://i.imgur.com/skFDqQq.png)

輸入可識別的名稱，並在ISO Image的地方選擇剛下載的Ubuntu

![](https://i.imgur.com/MwVdPAr.png)

![](https://i.imgur.com/8JlY602.png)

之後一直下一步到完成


## 第二步：環境搭建(舊版) 



在每個Node上都做設定，並且下載putty，以SSH連接。

[下載網址](https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html ) 

![](https://i.imgur.com/Ax7RMPP.png)

![](https://i.imgur.com/ybmxn7m.png)

![](https://i.imgur.com/szptP1f.png)

![](https://i.imgur.com/9hylssn.png)

![](https://i.imgur.com/XSGNRxo.png)

![](https://i.imgur.com/b34hBGU.png)

註：新版介面較難觀看，可參考舊版邏輯後運用到新版

使用NAT網路來模擬固定IP，透過SSH連線來操作Node以及開放服務的Port。
進入Virtual Box 喜好設定

![](https://i.imgur.com/f6jw0LP.png)

圖片中的【001】表示第一個集群，其他以此類推
這裡設定的目的是要讓實體本機與虛擬機能夠做IP轉換
以【001】k8s-m1為例子
連接到本機IP：127.0.0.1的2230port
會轉發到具有IP：10.0.2.13、Port22的虛擬機
透過這樣的機制可以把指定的連線透過轉換連接到虛擬機上
讓我們後續使用Putty連線可以方便操作虛擬機

主機連接埠從2230開始設定
虛擬機IP需要到虛擬機裡面查看，後續會說明
虛擬機Port皆為22
![](https://i.imgur.com/LxVFyyi.png)

## 第二步：環境搭建(新版)



在每個Node上都做設定，並且下載putty，以SSH連接。

[下載網址](https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html ) 


![](https://i.imgur.com/ybmxn7m.png)

![](https://i.imgur.com/szptP1f.png)

![](https://i.imgur.com/9hylssn.png)

![](https://i.imgur.com/XSGNRxo.png)

![](https://i.imgur.com/b34hBGU.png)

主機連接埠從2230開始設定
虛擬機IP需要到虛擬機裡面查看，後續會說明
虛擬機Port皆為22

![](https://i.imgur.com/fweIaWK.png)
## 第三步：Node設定(每台Control Plane、Worker Node都要)

安裝 net-tools
```=bash
sudo apt install net-tools
```

查詢IP
```=bash
ifconfig
```

關閉swap，K8S會需要使用到Swap memory
```=bash
sudo vim /etc/fstab (# 註解此行：/Swap.img …… # comment out it -> #/Swap.img …)

sudo vim /etc/sysctl.conf

vm.swappiness=0  (於最後面加入此行)

sudo shutdown -r now (更改後重啟)

free -m (測試是否成功)
```

Set hostname (每個Node都要，用於Node之間的通信)
```=bash
sudo vim /etc/hosts
```
將各台虛擬機的IP改好

10.0.2.13 k8s-m1
10.0.2.14 k8s-m2
10.0.2.16 k8s-m3
10.0.2.15 k8s-w1
10.0.2.17 k8s-w2
10.0.2.18 k8s-w3
10.0.2.19 k8s-w4

舉其中一台為例，這台的ＩＰ是10.0.2.5，如下圖
![](https://hackmd.io/_uploads/H1HuhDyD3.png)


Enable IPv4 Forwarding
```=bash
sudo modprobe overlay

sudo modprobe br_netfilter

echo "net.bridge.bridge-nf-call-iptables = 1" | sudo tee -a /etc/sysctl.d/99-kubernetes-cri.conf

echo "net.ipv4.ip_forward = 1" | sudo tee -a /etc/sysctl.d/99-kubernetes-cri.conf

echo "net.bridge.bridge-nf-call-ip6tables = 1" | sudo tee -a /etc/sysctl.d/99-kubernetes-cri.conf

sudo sysctl --system
```
Set cri-o versions to use
```=bash
export OS=xUbuntu_20.04

export VERSION=1.23

echo "deb http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/$VERSION/$OS/ /" | sudo tee -a /etc/apt/sources.list.d/cri-0.list

curl -L http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/$VERSION/$OS/Release.key | sudo apt-key add -

echo "deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/ /" | sudo tee -a /etc/apt/sources.list.d/libcontainers.list

sudo apt-get update
```
Install cri-o
```=bash
sudo apt-get install -y cri-o cri-o-runc podman buildah

sudo systemctl daemon-reload

sudo systemctl enable crio

sudo systemctl start crio

systemctl status crio (查看是否正在running)
```
Add Kubernetes repo and software
```
sudo sh -c "echo 'deb http://apt.kubernetes.io/ kubernetes-xenial main' >> /etc/apt/sources.list.d/kubernetes.list"

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

sudo apt-get update

sudo apt-get install -y kubeadm=1.23.4-00 kubelet=1.23.4-00 kubectl=1.23.4-00

sudo apt-mark hold kubeadm kubelet kubectl

sudo apt-get update
```

## 第四步：Control Plane 1 設定

生成 kube-vip Pod
```=bash
sudo su

export VIP=10.0.2.100

export INTERFACE=enp0s3

podman image pull docker.io/plndr/kube-vip:v0.3.8
```

```=bash
podman run --rm --network=host --entrypoint=/kube-vip ghcr.io/kube-vip/kube-vip:v0.3.8 \
  manifest pod \
  --interface $INTERFACE \
  --vip $VIP \
  --controlplane \
  --services \
  --arp \
  --leaderElection | tee  /etc/kubernetes/manifests/kube-vip.yaml
```

配置kubeadm-config 用於初始化集群
```=bash
sudo nano kubeadm-config.yaml
```

```=bash
apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
nodeRegistration:
  criSocket: "/var/run/crio/crio.sock"
  imagePullPolicy: IfNotPresent
---
kind: ClusterConfiguration
apiVersion: kubeadm.k8s.io/v1beta3
kubernetesVersion: v1.23.4
clusterName: kubernetes
certificatesDir: /etc/kubernetes/pki
imageRepository: k8s.gcr.io
controlPlaneEndpoint: "10.0.2.100:6443"  # 填 apiserver 的 vip 地址
networking:
  serviceSubnet: "10.96.0.0/16"
  podSubnet: "10.244.0.0/16"
etcd:
  local:
    dataDir: /var/lib/etcd
```
Init Cluster
```=bash
sudo kubeadm init --config=kubeadm-config.yaml --upload-certs
```
會跳出以下提示加入Master及Worker之Key (需記住key)
```=bash
Control Plane:
sudo  kubeadm join 10.0.2.100:6443 –token uw34sa.w1x32k3j7bhc98dt \
        --discovery-token-ca-cert-hash sha256:8be6cb64ab77c4015e5ad29fc1f501f3a21d0e2f55ceb9545e585c0cd3b4c747 \
        --control-plane –certificate-key d8a4e40c4a65af755fd59a9aed705e9cf5fcc9060232454fd6096b6ddb435415

Worker Node:
sudo kubeadm join 10.0.2.100:6443 --token uw34sa.w1x32k3j7bhc98dt \
        --discovery-token-ca-cert-hash sha256:8be6cb64ab77c4015e5ad29fc1f501f3a21d0e2f55ceb9545e585c0cd3b4c747

#Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
#As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
#"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.
```

```=bash
mkdir -p $HOME/.kube

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

sudo chown $(id -u):$(id -g) $HOME/.kube/config
```

## 第五步：Control Plane 2~n 設定(有兩個以上的Control Plane才要做)

```=bash
sudo su
```

Add Control Plane
```=bash
sudo  kubeadm join 10.0.2.100:6443 --token uw34sa.w1x32k3j7bhc98dt \
        --discovery-token-ca-cert-hash sha256:8be6cb64ab77c4015e5ad29fc1f501f3a21d0e2f55ceb9545e585c0cd3b4c747 \
        --control-plane --certificate-key d8a4e40c4a65af755fd59a9aed705e9cf5fcc9060232454fd6096b6ddb435415
```
```=bash
mkdir -p $HOME/.kube

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

sudo chown $(id -u):$(id -g) $HOME/.kube/config
```
測試是否可用
```=bash
kubectl get cs

kubectl get node
```

將加入之Control Plane生成Kube-vip
```=bash
sudo su

export VIP=10.0.2.100

export INTERFACE=enp0s3

podman image pull docker.io/plndr/kube-vip:v0.3.8
```

```=bash
podman run --rm --network=host --entrypoint=/kube-vip ghcr.io/kube-vip/kube-vip:v0.3.8 \
  manifest pod \
  --interface $INTERFACE \
  --vip $VIP \
  --controlplane \
  --services \
  --arp \
  --leaderElection | tee  /etc/kubernetes/manifests/kube-vip.yaml
```

```=bash
kubectl get pods -A | grep vip
```

## 第六步：Worker Node設定

加入Worker
```=bash
sudo kubeadm join 10.0.2.100:6443 --token uw34sa.w1x32k3j7bhc98dt \
        --discovery-token-ca-cert-hash sha256:8be6cb64ab77c4015e5ad29fc1f501f3a21d0e2f55ceb9545e585c0cd3b4c747
```

都加入後在Control Plane節點上確認是否都有加入成功 (在Control Plane上執行)
```=bash
kubectl get node (總共7個)
```
![](https://i.imgur.com/IL7KCg9.png)


## 第七步：網路元件設定(在Control Plane 1上執行)

配置calico yaml

```=bash
curl https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/calico.yaml -O

kubectl apply -f calico.yaml
```
![](https://i.imgur.com/ReND6QS.png)


確認calico pod 及kube-vip成功running(如下圖)
```
kubectl get pod -n=kube-system
```

測試高可用 (logs用來查看pod日誌)
```=bash
kubectl logs -f kube-vip-k8sha-m1 -n=kube-system
```

如果是leader會顯示如下
![](https://i.imgur.com/8LVZLLa.png)
其餘會顯示如下
![](https://i.imgur.com/ox0uBm6.png)


## 之後虛擬機的關閉記得都要選擇ACPI關機

![](https://hackmd.io/_uploads/rJRA_7xDn.png)

## (可略過)第八步：替node打上label(在Control Plane 1上執行)




下面指令的用意是，將worker node 1 打上 one 的標籤；將worker node 2 打上 two 的標籤，用來之後指定 wazuh master 到 worker node 1以及指令 wazuh worker 到 worker node 2上

```=bash
kubectl label nodes k8s-s3-w1 work_num=one

kubectl label nodes k8s-s3-w2 work_num=two
```

接續NFS安裝教學