# CentOS7 Highly Available K8S (3Master 1Worker)
## 基本設定
### 設置靜態IP
**```$ vi /etc/sysconfig/network-scripts/ipcfg-ens192```**
```bash=
TYPE="Ethernet"
PROXY_METHOD="none"
BROWSER_ONLY="no"
BOOTPROTO="static"
DEFROUTE="yes"
IPV4_FAILURE_FATAL="no"
IPV6INIT="yes"
IPV6_AUTOCONF="yes"
IPV6_DEFROUTE="yes"
IPV6_FAILURE_FATAL="no"
IPV6_ADDR_GEN_MODE="stable-privacy"
NAME="ens192"
UUID="0b905541-6f11-4eb4-a685-5777d4ab5f0e"
DEVICE="ens192"
ONBOOT="yes"
HWADDR="00:50:56:8B:F1:26"
MACADDR="preserve"
IPADDR="172.31.222.226"
PREFIX="24"
GATEWAY="172.31.222.1"
DNS1="8.8.8.8"https://hackmd.io/tenzByVQS-Cpti-pIdOZXw?both#
IPV6_PRIVACY="no"
```
**```$ ifdown ens192```**
**```$ ifup ens192```**
**```$ vi /etc/hosts```**
```bash=
192.168.222.221 master01
192.168.222.222 master02
192.168.222.223 master03
192.168.222.224 worker01
```
## 安裝K8S
### 安裝Docker
###### 參考網址https://stevenitlife.blogspot.com/2020/01/k8s-cluster.html
**更新套件**
**`$ yum update`**
(若舊已有Docker → 要刪光)
```
$ yum remove docker \
> docker-client \
> docker-client-latest \
> docker-common \
> docker-latest \
> docker-latest-logrotate \
> docker-logrotate \
> docker-engine
```
**安裝相依檔套件**
**``$ sudo yum install -y yum-utils``**
**``$ sudo yum-config-manager \
--add-repo \
https://download.docker.com/linux/centos/docker-ce.repo``**
**安裝最新版Docker**
**``$ yum install docker-ce docker-ce-cli containerd.io``**
**啟動Docker**
**``$ systemctl start docker``**
**Docker自動啟動**
**``$ systemctl enable docker``**
**編寫設定檔**
**```$ vi /etc/docker/daemon.json ```**
```bash=
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2",
"storage-opts": [
"overlay2.override_kernel_check=true"
]
}
```
**重啟Docker**
**``$ systemctl daemon-reload``**
**``$ systemctl restart docker``**
### 安裝套件及設定
**關SELinux**
**``$ setenforce 0``**
**``$ sed -i 's@SELINUX=enforcing@SELINUX=disabled@' /etc/sysconfig/selinux``**
(或是 **``$ vi /etc/sysconfig/selinux``** → **SELINUX=enforcing→disabled**)
**關Swap**
**``$ swapoff -a``**
**``$ vi /etc/fstab``** → **swap** 那行加上 **#** 註解掉
**關閉防火牆**
```bash=
$ systemctl disable firewalld && systemctl stop firewalld
$ echo 1 > /proc/sys/net/ipv4/ip_forward
$ echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf
$ echo "net.bridge.bridge-nf-call-iptables = 1" >> /etc/sysctl.conf
$ modprobe br_netfilter
$ echo "br_netfilter" > /etc/modules-load.d/br_netfilter.conf
$ sysctl -p
$ lsmod | grep br_netfilter
```
**安裝Kubectl kubeadm kubelet**
**``$ vi /etc/yum.repos.d/kubernetes.repo``**
```bash=
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
```
###### **更新repo list** ``yum repolist -y``
**``$ sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes``**
**編寫設定檔**
**``$ vi /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf``**
```bash=
# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/sysconfig/kubelet
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=systemd"
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_CGROUP_ARGS $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS
```
**啟用Kubelet**
**``$ systemctl start kubelet``**
**``$ systemctl enable kubelet``**
- **檢查**
**``$ systemctl status -l kubelet``**
### 設置Load-Balancer(master三台)
**安裝haproxy、keepalived**
**```$ yum install -y haproxy keepalived```**
**haproxy設定**
**```$ vi /etc/haproxy/haproxy.cfg```**
```bash=
#---------------------------------------------------------------------
# Example configuration for a possible web application. See the
# full configuration options online.
#
# http://haproxy.1wt.eu/download/1.4/doc/configuration.txt
#
#---------------------------------------------------------------------
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
# to have these messages end up in /var/log/haproxy.log you will
# need to:
#
# 1) configure syslog to accept network log events. This is done
# by adding the '-r' option to the SYSLOGD_OPTIONS in
# /etc/sysconfig/syslog
#
# 2) configure local2 events to go to the /var/log/haproxy.log
# file. A line like the following can be added to
# /etc/sysconfig/syslog
#
# local2.* /var/log/haproxy.log
#
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
# turn on stats unix socket
stats socket /var/lib/haproxy/stats
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
#---------------------------------------------------------------------
# kubernetes apiserver frontend which proxys to the backends
#---------------------------------------------------------------------
frontend kubernetes
mode tcp
bind *:16443
option tcplog
default_backend kubernetes-apiserver
#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend kubernetes-apiserver
mode tcp
option tcp-check
balance roundrobin
server master01 192.168.222.221:6443 check
server master02 192.168.222.222:6443 check
server master03 192.168.222.223:6443 check
```
**啟用haproxy**
**```$ systemctl start haproxy```**
**```$ systemctl enable haproxy```**
- **檢查**
**```$ systemctl status -l haproxy```**
**keepalived設定**
**```$ vi /etc/keepalived/keepalived.conf```**
- **MASTER01**
```bash=
! Configuration File for keepalived
global_defs {
router-id master01
}
vrrp_instance VI_1 {
state MASTER
interface ens192
virtual_router_id 51
priority 150
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.222.220 #虛擬IP : VIP
}
}
```
- MASTER02
```bash=
! Configuration File for keepalived
global_defs {
router-id master02 #主機名
}
vrrp_instance VI_1 {
state BACKUP #改成備援
interface ens192
virtual_router_id 51
priority 125 #順序改比master01低
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.222.220
}
}
```
- MASTER03
```bash=
! Configuration File for keepalived
global_defs {
router-id master03
}
vrrp_instance VI_1 {
state BACKUP
interface ens192
virtual_router_id 51
priority 100 #順序改比master2低
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.222.220
}
}
```
**啟用keepalived**
**```$ systemctl start keepalived```**
**```$ systemctl enable keepalived```**
- **檢查**
**```$ systemctl status -l keepalived```**
### 設定K8S叢集
**kubeadm 初始化**
**``$ sudo kubeadm init --control-plane-endpoint "192.168.222.220:6443" --upload-certs --service-cidr 10.98.0.0/24 --pod-network-cidr 10.244.0.0/16 --apiserver-advertise-address 192.168.222.221``**
**建K8S要求設定**
**``$ mkdir –p $HOME/.kube``
``$ cp -i /etc/kubernetes/admin.conf $HOME/.kube/config``
``$ chown $(id –u):$(id –g) $HOME/.kube/config``**
### 節點加入
**Worker機加入**
**``$ kubeadm join 192.168.222.220:6443 --token 10b85f.3b50ysmx9b8kyi1e \
--discovery-token-ca-cert-hash sha256:f9defe868f080eeaf961628a0497cac66b3ebb1927cd1e59e2d85035da74f757 ``**
**Master機加入**
**``$ kubeadm join 192.168.222.220:6443 --token 10b85f.3b50ysmx9b8kyi1e \
--discovery-token-ca-cert-hash sha256:f9defe868f080eeaf961628a0497cac66b3ebb1927cd1e59e2d85035da74f757 \
--control-plane --certificate-key 9772574e7ca3de9f9d8ed5284d00d02e7948faeec78d32f5f23ce4b1a5277182``**
**加入CNI - Flannel**
**``$ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml``**
**設定所有Master可以兼當Worker(pod生成在Master上)**
**``$ kubectl taint nodes –all node-role.kubernetes.io/master-
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/``**
**將設定檔複製給Master Node**
**```scp .kube 192.168.222.222:~/```**
{"metaMigratedAt":"2023-06-16T19:42:02.681Z","metaMigratedFrom":"Content","title":"CentOS7 Highly Available K8S (3Master 1Worker)","breaks":true,"description":"$ vi /etc/sysconfig/network-scripts/ipcfg-ens192","showTags":"false","contributors":"[{\"id\":\"e823fd16-040d-4607-9480-63873bec7fff\",\"add\":10754,\"del\":626}]"}