---
# System prepended metadata

title: 使用3台vagrant建立的VMs架設一個k8s叢集
tags: [kubernetes, docker, vagrant, k8s]

---

使用3台vagrant建立的VMs架設一個k8s叢集
===
![](https://i.imgur.com/K3vA7rD.png)

## 建立一個3台機器的`Vagrantfile`
```ruby=
Vagrant.configure("2") do |config|
  config.vm.define "master" do |master|
    master.vm.box = "ubuntu/bionic64"
    master.vm.network "private_network", ip: "192.168.2.2"
    master.vm.provider "virtualbox" do |vb|
      vb.memory = "16384"
    end
  end
  config.vm.define "node1" do |node1|
    node1.vm.box = "ubuntu/bionic64"
    node1.vm.network "private_network", ip: "192.168.2.3"
    node1.vm.provider "virtualbox" do |vb|
      vb.memory = "8192"
    end
  end
  config.vm.define "node2" do |node2|
    node2.vm.box = "ubuntu/bionic64"
    node2.vm.network "private_network", ip: "192.168.2.4"
    node2.vm.provider "virtualbox" do |vb|
      vb.memory = "8192"
    end
  end
end
```
## 三台主機基本設定
* 設定主機名稱
```
sudo hostnamectl set-hostname master
sudo hostnamectl set-hostname node1
sudo hostnamectl set-hostname node2
```
* 設定`dns`
```
# /etc/hosts
192.168.2.2 master 
192.168.2.3 node1 
192.168.2.4 node2
```
* 安裝必要套件
```
sudo apt update -y
sudo apt upgrade -y
sudo apt install linux-image-extra-virtual -y
sudo reboot
```

* 一定要將swap關掉
```
sudo swapoff -a
```
* 建立`k8s-admin`這個管理k8s的使用者
```
sudo useradd -s /bin/bash -m k8s-admin
sudo passwd k8s-admin
sudo usermod -aG sudo k8s-admin
echo "k8s-admin ALL=(ALL) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/k8s-admin
```
## 安裝`docker`作為k8s的runtime
* 安裝`docker engine`
```
sudo apt-get update -y
sudo apt-get upgrade -y
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
sudo usermod -aG docker k8s-admin
sudo usermod -aG docker $USER
sudo reboot
```
* 將`docker engine`設定成標準的服務(**這步一定要做！**)
```
sudo mkdir /etc/docker
cat <<EOF | sudo tee /etc/docker/daemon.json
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
EOF

sudo systemctl enable docker
sudo systemctl daemon-reload
sudo systemctl restart docker
sudo reboot
```
## 安裝k8s
* 加入`apt`源
```
# /etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
```
* 獲取`google`的金鑰
```
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
```
* 安裝必要的k8s元件
```
sudo apt update -y
sudo apt install kubectl kubelet kubeadm kubernetes-cni -y
```
* 確定k8s的三個元件都安裝好了
```
$ which kubelet
/usr/bin/kubelet
$ which kubeadm
/usr/bin/kubeadm
sudo reboot
```
* 設定環境變數
```
export API_ADDR="192.168.2.2"
export DNS_DOMAIN="k8s.local"
export POD_NET="10.4.0.0/16"
export SRV_NET="10.5.0.0/16"
```
* 建立k8s叢集
```
sudo kubeadm init --pod-network-cidr ${POD_NET} --service-cidr ${SRV_NET} \
--service-dns-domain "${DNS_DOMAIN}" --apiserver-advertise-address ${API_ADDR}
```
![](https://i.imgur.com/Vby5fPK.png)
**！！圖中產生的指令一定要記下來，要不然要重新產生！！**

* 複製設定檔，從這步開始一定要用`k8s-admin`這個使用者
```
su - k8s-admin
mkdir -p $HOME/.k8s
sudo cp -i /etc/kubernetes/admin.conf $HOME/.k8s/config
sudo chown $(id -u):$(id -g) $HOME/.k8s/config
export KUBECONFIG=$HOME/.k8s/config
echo "export KUBECONFIG=$HOME/.k8s/config" | tee -a ~/.bashrc
```
* 安裝`weave`
```
# su - k8s-admin
$ kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"
```
* 產生`join`的指令
```
kubeadm token create --print-join-command

sudo kubeadm join 192.168.2.2:6443 --token 1lg6hx.4kjnjwtk9axkuh33 --discovery-token-ca-cert-hash sha256:94b0450336effe7b97a3c70aac267d9866bcd1a6911f438f0d571effcdb6061d
```

## node加入cluster
* 加入cluster，首先要先`ssh`進入node機器，並且安裝好`kubectl` `kubelet` `kubeadm` `kubernetes-cni`這一步為止。
```
sudo kubeadm join 192.168.2.2:6443 --token 1lg6hx.4kjnjwtk9axkuh33 --discovery-token-ca-cert-hash sha256:94b0450336effe7b97a3c70aac267d9866bcd1a6911f438f0d571effcdb6061d
```

* 加入後確定k8s有跑起來
```
sudo systemctl status kubelet
```
![](https://i.imgur.com/kwzZ0Ad.png)

## 控制cluster
* 回到`master`主機上，切換使用者到`k8s-admin`，查看已經加入的nodes
```
vagrant@master:~$su - k8s-admin
k8s-admin@master:~$ kubectl get nodes
NAME     STATUS   ROLES                  AGE   VERSION
master   Ready    control-plane,master   9h    v1.22.3
node1    Ready    <none>                 9h    v1.22.3
node2    Ready    <none>                 9h    v1.22.3
k8s-admin@master:~$
```
* 建立一個網頁伺服器的`deployment`，檔名舉例為`nginx.yaml`
```yaml=
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 3 # tells deployment to run 3 pods matching the template
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        # image: nginx:latest
        image: registry.hub.docker.com/joshhu/test-php-site
        ports:
        - containerPort: 80
```
* 啟動這個`deployment`，就會啟動`pods`
```
kubectl apply -f nginx.yaml
```
* 查看`deployments`，也可查看`pods`
```
kubectl get deployments
kubectl get pods
```
* 查看`deployment`詳情
```
k8s-admin@master:~$ kubectl describe deployment nginx-deployment
Name:                   nginx-deployment
Namespace:              default
CreationTimestamp:      Mon, 01 Nov 2021 17:33:47 +0000
Labels:                 <none>
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               app=nginx
Replicas:               3 desired | 3 updated | 3 total | 3 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=nginx
  Containers:
   nginx:
    Image:        registry.hub.docker.com/joshhu/test-php-site
    Port:         80/TCP
    Host Port:    0/TCP
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   nginx-deployment-d75ccb (3/3 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  55m   deployment-controller  Scaled up replica set nginx-deployment-d75ccb to 3
k8s-admin@master:~$
```
## 建立一個對外開放的`service`，檔名為`nginx-service.yaml`
* 建立一個`service`物件如下
```yaml=
apiVersion: v1
kind: Service
metadata:
  name: nginx-svc
  labels:
    app: nginx
spec:
  type: NodePort
  ports:
  - port: 80
    nodePort: 30080
  selector:
    app: nginx
  externalIPs:
  - 192.168.2.3
```
* 啟動`service`
```
kubectl apply -f nginx-service.yaml
```
* 查看這個`service`
```
k8s-admin@master:~$ kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP   10.5.0.1     <none>        443/TCP        10h
nginx-svc    NodePort    10.5.0.187   192.168.2.3   80:30080/TCP   58m
k8s-admin@master:~$
```
* `service`詳細資料
```
k8s-admin@master:~$ kubectl describe svc nginx-svc
Name:                     nginx-svc
Namespace:                default
Labels:                   app=nginx
Annotations:              <none>
Selector:                 app=nginx
Type:                     NodePort
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.5.0.187
IPs:                      10.5.0.187
External IPs:             192.168.2.3
Port:                     <unset>  80/TCP
TargetPort:               80/TCP
NodePort:                 <unset>  30080/TCP
Endpoints:                10.36.0.1:80,10.44.0.1:80,10.44.0.2:80
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>
k8s-admin@master:~$
```
* 網頁瀏覽
![](https://i.imgur.com/dbe05eq.jpg)

## 參考資料
https://computingforgeeks.com/how-to-setup-3-node-kubernetes-cluster-on-ubuntu-18-04-with-weave-net-cni/

https://gist.github.com/danielepolencic/ef4ddb763fd9a18bf2f1eaaa2e337544

###### tags: `k8s`, `kubernetes`, `docker`, `vagrant`