--- tags: linux,build --- <style> @keyframes bbb {/*轉圈效果*/ 0% { transform: rotate(0deg) scale(1.5, 1.5); } 50% { transform: rotate(180deg) scale(1.5, 1.5); } 100% { transform: rotate(360deg) scale(1.5, 1.5); } } .ui-user-icon/*變更頭貼*/, .ui-avatar/*頭貼*/, .fa-plus/*新增文件符號*/ { animation: bbb .75s linear infinite; } </style> > [name=Nick Chang] 因ubuntu 22.04問題，需調整nameserver設定 ``` vim /etc/systemd/resolved.conf rm /etc/resolv.conf systemctl daemon-reload && systemctl restart systemd-resolved.service && systemctl status -l systemd-resolved.service --no-pager ln -s -f /run/systemd/resolve/resolv.conf /etc/resolv.conf ``` ### 1.Kubelet / Kubeadm 安裝與概念 於欲執行 Kubernetes 的機上，必須要有 kubelet / kubeadm ( Master 與 node 皆要執行此步驟 ) 這是 Kubernetes 的主要程式，然後透過 runtime 來實現 pods 這裡的 runtime 可以是 Docker / CRI-O / Containerd #### a. 先設定 k8s server上網路 ``` cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 EOF sudo sysctl --system ``` #### b. 安裝 kubeadm / kubelet ``` sudo apt-get update && sudo apt-get install -y apt-transport-https curl curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list deb https://apt.kubernetes.io/ kubernetes-xenial main EOF sudo apt-get update sudo apt-get install -y kubelet kubeadm kubectl sudo apt-mark hold kubelet kubeadm kubectl ``` #### c. 驗證 只要輸入 kubeadm 就可以知道是否安裝成功 ( 如下畫面 ) ![](https://i.imgur.com/5Fj8iwI.png) ### 2. 安裝 CRI-O for K8s CRI-O 就是設計給 Kubernetes 的 Container Runtime Interface，k8s 當然也可以使用 Docker 作為 Container Runtime，但是我們這邊遵循選擇 cri-o 為主 ( cri-o github 於此 : https://github.com/cri-o/cri-o ) 本步驟全部以 Root 身份執行 #### a. 先設定 Linux kernel 模組 ``` modprobe overlay modprobe br_netfilter ``` #### b. 設定網路給 CRI-O ``` cat > /etc/sysctl.d/99-kubernetes-cri.conf <<EOF net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 net.bridge.bridge-nf-call-ip6tables = 1 EOF sysctl --system ``` #### c. 安裝 CRI-O 請注意這邊的 VERSION 與 OS，如果你選擇非本教學的版本請自行修改 詳細參數可以在官網看到 ``` export OS=xUbuntu_22.04 export CRIO_VERSION=1.24 echo "deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/ /"|sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list echo "deb http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/$CRIO_VERSION/$OS/ /"|sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable:cri-o:$CRIO_VERSION.list curl -L https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:$CRIO_VERSION/$OS/Release.key | sudo apt-key add - curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/Release.key | sudo apt-key add - sudo apt update sudo apt info cri-o sudo apt install cri-o cri-o-runc sudo systemctl start crio sudo systemctl enable crio sudo systemctl status crio ``` 在大多的官方、網友教學文章中都是以 docker 作為 Container Runtime Interface， 但是這邊選擇 CRI-O，故在 cgroup 部分需要特別指定。 預設是 docker cgroupfs，可以不需特別指定，CRI-O 需要指定 systemd。 ``` #查看 CRI-O 的 cgroup driver cat /etc/crio/crio.conf | grep cgroup_manager # cgroup_manager is the cgroup management implementation to be used cgroup_manager = "systemd" ``` 設定 cgroup-driver & ExecStart 增加 $KUBELET_CGROUP_ARGS ``` vim /etc/default/kubelet.yaml ``` ```bash= apiVersion: kubelet.config.k8s.io/v1beta1 cgroupDriver: systemd containerRuntimeEndpoint: 'unix:///var/run/crio/crio.sock' kind: KubeletConfiguration runtimeRequestTimeout: 5m ``` ``` sudo vi /etc/systemd/system/kubelet.service.d/10-kubeadm.conf ``` ```bash= Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=systemd" #EnvironmentFile=-/etc/default/kubelet EnvironmentFile=-/etc/default/kubelet.yaml ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS $KUBELET_CGROUP_ARGS ``` 因 kubelet config file 有異動，必須進行 reload & restart kubelet ``` sudo systemctl daemon-reload sudo systemctl restart kubelet ``` #### e. 關閉 swap K8S 預設不希望我們系統有 swap 存在，所以 : 編輯 /etc/fstab ( 執行 sudo vim /etc/fstab )，將 swap 註解掉( 下行 ) ```bash= # /swap.img none swap sw 0 0 ``` 然後執行 Ubuntu 關閉 Swap 指令 ``` sudo swapoff -a ``` ### 3. 設定服務自動重啟 ( 本步驟全部以 Root 身份執行，後略 ) 我們要給 kubelet 預設的設定 ( /etc/default/kubelet ) ``` cat > /etc/default/kubelet <<EOF KUBELET_EXTRA_ARGS=--feature-gates="AllAlpha=false,RunAsGroup=true" --container-runtime=remote --cgroup-driver=systemd --container-runtime-endpoint='unix:///var/run/crio/crio.sock' --runtime-request-timeout=5m EOF ``` 還有我們需要這個流程，確保所有的服務在重開機後會自己起來 ( k8s reboot 後依然有效 ) ``` systemctl enable kubelet systemctl enable crio ``` ### 4. K8s Master Node 啟動 : kubeadm init 如果你是使用 VM，可以選擇在這步驟完成之前先 snapshot / clone ，或是在另外一台 Server 上執行以上安裝步驟 因為我們要建立 master node 了，其實相當簡單，就是輸入 ( 真相不是以下指令這麼簡單，先看完此步驟全文再輸入，或是參考此步驟小總結 ) ``` kubeadm init ``` 如果成功則會看到以下資訊 ![](https://i.imgur.com/V3CcWCV.png) ### 5. K8s Master Node 加入 Cluster : kubeadm join 在加入之前需要另外apply k8s控制層網路flannel，需要在Master Node執行以下指令 ``` kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml ``` 然後需要刪除原有網卡設定會需要重置 ``` kubeadm reset rm /etc/cni/net.d/100-crio-bridge.conf ifconfig cni0 down ip link del cni0 systemctl restart crio systemctl restart kubelet export CIDR=10.244.0.0/16 kubeadm init --pod-network-cidr=$CIDR --cri-socket=/var/run/crio/crio.sock kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml ``` 直接在想加入的node上輸入上面join的指令即可 ``` kubeadm join 192.168.250.50:6443 --token 2zpajs.x6ozx7c4tegnsij8 \ --discovery-token-ca-cert-hash sha256:ba62b5f8583f0315d02b395c639cc2a3f75c3d621e389dcd659a727d4a88259d ``` 如果token過期可以用下面這組指令再次建立 ``` kubeadm token create --print-join-command ``` ### 6.開始操作K8s 若想要操作k8s，則需要在使用者目錄創建.kube資料夾，並在資料夾下具有admin.conf檔案 ``` mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config ``` #Alternatively, if you are the root user, you can run: ``` export KUBECONFIG=/etc/kubernetes/admin.conf ``` # 狀況處理 ### 狀況1 — 機器上 Docker 同時存在，導致 kubeadm 分不清楚是要用 docker 還是 cri-0 錯誤訊息如下 : :::danger Found multiple CRI sockets, please use --cri-socket to select one: /var/run/dockershim.sock, /var/run/crio/crio.sock To see the stack trace of this error execute with --v=5 or higher ::: 解決辦法: 指定我們使用 crio 來執行 ``` kubeadm init --cri-socket=/var/run/crio/crio.sock ``` ### 狀況2 — 機器 swap 沒關 預設情況下 k8s 機器不能有 swap 存在 錯誤訊息如下: :::danger W0920 04:02:50.683554 7537 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io] [init] Using Kubernetes version: v1.19.2 [preflight] Running pre-flight checks error execution phase preflight: [preflight] Some fatal errors occurred: [ERROR Swap]: running with swap on is not supported. Please disable swap [preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...` To see the stack trace of this error execute with --v=5 or higher ::: 解決辦法: 關閉 swap 編輯 /etc/fstab ( 執行 sudo vim /etc/fstab )，將 swap 註解掉( 下行 ) ```bash= # /swap.img none swap sw 0 0 ``` 然後執行 Ubuntu 關閉 Swap 指令 ``` sudo swapoff -a ``` 然後再一次嘗試啟動 ``` kubeadm init --cri-socket=/var/run/crio/crio.sock ``` ### 狀況3 — Timeout 其實這是因為網路設定不正確導致 kubeadm 與 kubelet 無法驗證 錯誤訊息主要類似 Unfortunately, an error has occurred: timed out waiting for the condition : :::danger [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [kubelet-check] Initial timeout of 40s passed. Unfortunately, an error has occurred: timed out waiting for the condition This error is likely caused by: - The kubelet is not running - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled) If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands: - 'systemctl status kubelet' - 'journalctl -xeu kubelet' Additionally, a control plane component may have crashed or exited when started by the container runtime. To troubleshoot, list all containers using your preferred container runtimes CLI. Here is one example how you may list all Kubernetes containers running in cri-o/containerd using crictl: - 'crictl --runtime-endpoint /var/run/crio/crio.sock ps -a | grep kube | grep -v pause' Once you have found the failing container, you can inspect its logs with: - 'crictl --runtime-endpoint /var/run/crio/crio.sock logs CONTAINERID' error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster To see the stack trace of this error execute with --v=5 or higher ::: 解決辦法: 先看一下 crio-bridge 的網路設定 ( 執行 cat /etc/cni/net.d/100-crio-bridge.conf ) 應該會類似如下 ```bash= { "cniVersion": "0.3.1", "name": "crio", "type": "bridge", "bridge": "cni0", "isGateway": true, "ipMasq": true, "hairpinMode": true, "ipam": { "type": "host-local", "routes": [ { "dst": "0.0.0.0/0" }, { "dst": "1100:200::1/24" } ], "ranges": [ [{ "subnet": "10.244.0.0/16" }], [{ "subnet": "1100:200::/24" }] ] } } ``` 注意上面粗斜體的部分，這邊是 crio 的網路位置，一般來說大家應該都是 10.85.0.0/16 ，如果不一樣的話請自行修改以下指令 用以下指令啟動 ``` export CIDR=10.244.0.0/16 kubeadm init \ --pod-network-cidr=$CIDR \ --cri-socket=/var/run/crio/crio.sock ``` ### 狀況 4 — 已經存在 錯誤訊息類似 : ERROR FileAvailable — etc-kubernetes-manifests-kube-apiserver.yaml /etc/kubernetes/manifests/kube-apiserver.yaml already exists :::danger W0920 04:18:45.722186 14058 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io] [init] Using Kubernetes version: v1.19.2 [preflight] Running pre-flight checks error execution phase preflight: [preflight] Some fatal errors occurred: [ERROR FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists [ERROR FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml]: /etc/kubernetes/manifests/kube-controller-manager.yaml already exists [ERROR FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml]: /etc/kubernetes/manifests/kube-scheduler.yaml already exists [ERROR FileAvailable--etc-kubernetes-manifests-etcd.yaml]: /etc/kubernetes/manifests/etcd.yaml already exists [ERROR Port-10250]: Port 10250 is in use [preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...` To see the stack trace of this error execute with --v=5 or higher ::: 解決辦法 : 輸入指令清除 ``` kubeadm reset --cri-socket=/var/run/crio/crio.sock ``` ### 小總結 要避開以上狀況，請先執行狀況 2 ( 關閉 swap ) ，然後狀況 3 設定網路 理論上就可以執行以下指令 ``` export CIDR=10.244.0.0/16 kubeadm init \ --pod-network-cidr=$CIDR \ --cri-socket=/var/run/crio/crio.sock ``` 成功的話應該會看到類似以下畫面 ![](https://i.imgur.com/67Gprf5.png)
Last changed by  
 
言靈
226

Read more

Drone架設

[name=Nick Chang] 本次在ubuntu 22.04 使用Docker架設 Docker Compose Yaml version: '2' services: drone-server: image: drone/drone:latest

10/24/2023
Terminal 版面 (P10k)

[name=Nick Chang] 官方Github 安裝順序 安裝 zsh 安裝 oh-my-zsh 安裝 powerline font 安裝 zsh theme: powerlevel10k

7/20/2023
Docker Registry

[name=Nick Chang] 官方教學 透過Goddady購買域名後，將所有crt pem塞入同一檔案內 cat *.crt *.pem > mix.crt 接著因為docker在引用憑證時會有編碼問題，所以需要將crt和key的檔案第一行新增空格行 vim mix.crt

5/3/2023
架設伺服器SOP

[name=佳佑] 以下是使用ubuntu16.04 LTS建置 iptable 清除本機防火牆設定 $ iptables -F #清除所有已訂規則

4/17/2023
Read more from 言靈
Published on HackMD