Deployment Guide: Web Server with Kubernetes Load Balancing and Two Worker Nodes

:::success These instructions provide guidance for deploying a web server with two worker nodes and load balancing managed by Kubernetes. ::: Kubernetes installation --- To install Kubernetes (k8s) on Ubuntu, you can follow these steps: Update your system: ``` sudo apt update sudo apt upgrade -y ``` Disable swap memory: ``` sudo swapoff -a sudo sed -i '/ swap / s/^$.*$$/#\1/g' /etc/fstab ``` Reboot after modify the /etc/fstab config. ``` sudo reboot ``` Make sure your 6443 port is not used (k8s use 6443 as default). ``` nc -v 127.0.0.1 6443 nc: connect to 127.0.0.1 port 6443 (tcp) failed: Connection refused ``` Install Docker: ``` sudo apt-get update sudo apt-get install \ ca-certificates \ curl \ gnupg \ lsb-release ``` ``` sudo mkdir -p /etc/apt/keyrings curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg echo \ "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \ $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null ``` ``` sudo apt-get update sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin ``` Start docker ``` sudo service docker start ``` check docker status ``sudo docker info`` Add your user name to docker user group so you can use ``docker info`` without sudo. ``` sudo groupadd docker sudo usermod -aG docker $USER ``` Re-login or reboot to take effect. These instructions are copied from https://ithelp.ithome.com.tw/articles/10291081. Set docker Cgroup driver to systemd. ``` sudo bash -c "cat > /etc/docker/daemon.json <<EOF { \"exec-opts\": [\"native.cgroupdriver=systemd\"] } EOF " ``` Restart docker and check status. ``` sudo systemctl restart docker docker info | grep Cgroup ``` Enable and start the Docker service: ``` sudo systemctl enable docker sudo systemctl start docker ``` You should see ``` Cgroup Driver: systemd Cgroup Version: 2 ``` Install cri-dockerd (for ubuntu 22.04 jammy) ``` wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.4/cri-dockerd_0.3.4.3-0.ubuntu-jammy_amd64.deb sudo dpkg -i cri-dockerd_0.3.4.3-0.ubuntu-jammy_amd64.deb ``` Check if the installation is successful. ``` cri-dockerd --help ``` Reload service and check status. ``` sudo systemctl daemon-reload sudo systemctl enable cri-docker.service sudo systemctl enable --now cri-docker.socket sudo systemctl status cri-docker.socket ``` Start install k8s. Download the Kubernetes apt repository key: ``` curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | gpg --dearmor | sudo tee /usr/share/keyrings/kubernetes-archive-keyring.gpg >/dev/null ``` Add the Kubernetes apt repository: ``` echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list ``` Update repository ``sudo apt-get update`` Install Kubernetes components (kubelet, kubeadm, kubectl): ``` sudo apt-get install -y kubelet kubeadm kubectl sudo apt-mark hold kubelet kubeadm kubectl ``` # Follow this article to set up your k8s cluster. https://ithelp.ithome.com.tw/articles/10291343 ## kubeadm init on master node sudo kubeadm init \ --apiserver-advertise-address=192.168.1.221 \ --pod-network-cidr=10.244.0.0/16 \ --cri-socket /var/run/cri-dockerd.sock \ --service-cidr=10.96.0.0/12 ``` 當你的主機網段是 192.168.1.x，你必須確保 Kubernetes 使用的網段不與它衝突。以下是基於你的新網段的建議設定值： apiserver-advertise-address：這應該是你主機的 IP 地址。在這種情況下，它應該是 192.168.1.221。 pod-network-cidr：為你的 pods 選擇一個不同的網段。如果你的主機在 192.168.1.x，那麼你可能希望選擇像 10.244.0.0/16 這樣的完全不同的 CIDR。 service-cidr：這是用於 Kubernetes 服務的 IP 地址。選擇一個不同的 CIDR，例如 10.96.0.0/12。 ``` wget https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml vim kube-flannel.yml and make sure the Network CIDR is same as pod-network-cidr you used in kubeadm init. ```json net-conf.json: | { "Network": "10.244.0.0/16", "Backend": { "Type": "vxlan" } } ``` kubectl apply -f kube-flannel.yml Check the flannel pods is running. ``` ubuntu@ubuntu:~/app$ kubectl get pods -n kube-flannel NAME READY STATUS RESTARTS AGE kube-flannel-ds-qfjgx 1/1 Running 0 2m5s ``` # Deploy nginx-controller ``` kubectl create namespace ingress-nginx wget -O deploy-ingress-nginx.yaml https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.0.0/deploy/static/provider/cloud/deploy.yaml kubectl apply -f deploy-ingress-nginx.yaml ``` kubectl get services -n ingress-nginx ``` NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ingress-nginx-controller LoadBalancer 10.105.132.4 <pending> 80:30388/TCP,443:30966/TCP 16s ingress-nginx-controller-admission ClusterIP 10.100.138.171 <none> 443/TCP 16s ``` You need to fix the external-ip pending problem if you're not in a cloud enviroment. To make LoadBalancer (ingress-nginx) work, you must patch the externalIP. ``` kubectl patch svc <svc-name> -n <namespace> -p '{"spec": {"type": "LoadBalancer", "externalIPs":["192.168.1.221"]}}' ``` for example ``` kubectl patch svc ingress-nginx-controller -n ingress-nginx -p '{"spec": {"type": "LoadBalancer", "externalIPs":["192.168.1.221"]}}' ``` # Important! If you want to run pods on master node, must remove kubectl taint nodes <NODE_NAME> node-role.kubernetes.io/control-plane:NoSchedule- use kubectl get nodes to get <NODE_NAME>. kubectl taint nodes ddbrothers node-role.kubernetes.io/control-plane:NoSchedule- # Deploy command for bruce: ## gpt backend ### create namespace kcl create namespace gpt ### deploy pv make sure the path in yaml exists e.g."/data/k8s-pv/gpt-postgresql" and run : kubectl apply -f /home/ubuntu/k8s_config/gpt-pv.yaml kubectl get pv # should see the status of pv is `Available` or `Bound` If the pv status is not good, check the nodeAffinity setting in gpt-pv.yaml, the nodeAffinity may configured to deploy the pv to a specify node. If you have to delete PV or PVC and re apply, make sure you have a backup data. ### deploy db and web api #### 視情況註解yaml中的node selector kubectl apply -f ./DB/deployment.yaml kubectl apply -f ./Deployment/deployment.yaml ### use port forwarding to test the db and api kubectl port-forward pods/db-658785646d-rlcqv 35432:5432 -n gpt kubectl port-forward deployments/gpt-backend 8000:8000 -n gpt ## ddb kcl create namespace ddb ### deploy pv make sure the path in yaml exists e.g.`/data/k8s-pv/ddb-postgresql` and `/data/k8s-pv/ddb-odoo` and run : kubectl apply -f /home/ubuntu/k8s_config/ddb-pv.yaml kubectl get pv # should see the status of pv is `Available` or `Bound` ### label the node to match yaml file By default, the ddb service use nodeSelector below: nodeSelector: node-key: node-ddb Make sure the desired node is labeled to match the node selector. #### use this command kubectl label nodes ddbrothers node-key=node-ddb ### deploy the config map #### check the settings in the config map and then apply kubectl apply -f /home/ubuntu/app/odoo-ddb/configMap.yaml #### if the config map is not exists, find a configMap.example.yaml at the same folder or git repo. ### deploy db and web api kubectl apply -f /home/ubuntu/app/odoo-ddb/deployment.yaml kubectl get pv # should see the status of pv is `Available` or `Bound` ### use port forwarding to test the db and api kubectl port-forward pods/db-658785646d-rlcqv 35432:5432 -n gpt kubectl port-forward deployments/gpt-backend 8000:8000 -n gpt ## Let's Encrypt ### 安裝 CustomResourceDefinitions cd ~/k8s_config mkdir cert-manager cd cert-manager wget https://github.com/jetstack/cert-manager/releases/download/v1.7.1/cert-manager.crds.yaml kubectl apply -f cert-manager.crds.yaml ### 創建 namespace kubectl create namespace cert-manager ### 部署 cert-manager wget https://github.com/jetstack/cert-manager/releases/download/v1.7.1/cert-manager.yaml kubectl apply -f cert-manager.yaml ### 設置 Let's Encrypt 的 ClusterIssuer #### 建立issuer 這個檔案應該在github k8s_config repo 但要檢查其中的email設定 kubectl apply -f ./letsencrypt-issuer.yaml #### 建立每一個site專用的憑證這個檔案應該在github k8s_config repo kubectl apply -f birdeo-certs.yaml #### 重新部屬 gpt web api 其中的ingress tls設定的secretName必須和前面birdeo-certs.yaml中設定的相同 kubectl apply -f /home/ubuntu/app/project-gpt-backend/Deployment/deployment.yaml #### 重新部屬 ddb web api # 修改gpt-backend部屬的node到birdeonode2 ### 停止gpt namespace下的所有服務 kubectl scale deployment gpt-backend --replicas=0 -n gpt kubectl scale deployment db --replicas=0 -n gpt ### 備份pv對應的folder sudo cp -r /data/k8s-pv/gpt-postgresql /path/to/backup/dest ### 刪除pvc和pv kcl delete pvc postgres-data-pvc kcl delete pv postgresql-gpt-pv ### 將node2加入cluster 進入node2, `ssh ubuntu@192.168.1.222` 使用kubeadm join將node2加入cluster之後 ### 將birdeonode2 標記為node-key = birdeonode2 `kcl label nodes birdeonode2 node-key=birdeonode2` ### 設定pv部屬在birdeonode2 設定gpt-pv.yaml的nodeAffinity, 使pv部屬在node 2 ``` storageClassName: local-storage local: path: /data/k8s-pv/gpt-postgresql nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: node-key operator: In values: - birdeonode2 ``` #### 部屬pv `kcl apply -f /home/ubuntu/k8s_config/gpt-pv.yaml` 同樣用 `kcl get pv` 檢查 pv 狀態是否 Available ### DB:設定pvc and service 部屬在 birdeonode2 修改 /home/ubuntu/app/project-gpt-backend/DB/deployment.yaml Deployment增加 ``` spec: template: spec: nodeSelector: node-key: birdeonode2 ``` pvc根據名稱使用指定的pv不需要設定nodeSelector ``` spec: volumeName: postgresql-gpt-pv ``` 修改後apply即可 `kcl apply -f /home/ubuntu/app/project-gpt-backend/DB/deployment.yaml` 檢查 pod是否部屬到 birdeonode2 `kcl get pods -o wide` ### API:設定部屬到birdeonode2 同DB設定修改yaml ### 設定node selector vim /home/ubuntu/app/project-gpt-backend/DB/deployment.yaml 設定其中的 node selector ``` spec: nodeSelector: node-key: birdeonode2 ``` # [STOP] The following instruction is WRONG Initialize the Kubernetes cluster on the **master node**: ``` sudo kubeadm init ``` **Note down the kubeadm join command** that is displayed at the end of the initialization process. You will need it to join worker nodes to the cluster. Set up the Kubernetes configuration for the current user: ``` mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config ``` Install the pod network add-on. For example, you can use Calico: ``` kubectl apply -f https://docs.projectcalico.org/v3.16/manifests/calico.yaml ``` Join worker nodes to the cluster: On each worker node, run the kubeadm join command that you noted down in step 8. Verify the cluster status: ``` kubectl get nodes ``` You should see the master node and the joined worker nodes in the output. That's it! You have successfully installed Kubernetes on Ubuntu. K8s service configuration --- If you have only two EC2 instances and want to set up a simple load balancing strategy between them using Kubernetes, you can use a single LoadBalancer Service to distribute traffic to the two instances. Here's an example of how you can set it up: Create a YAML file for the LoadBalancer Service. Let's name it loadbalancer.yaml: ``` apiVersion: v1 kind: Service metadata: name: my-load-balancer spec: type: LoadBalancer selector: app: my-web-app ports: - protocol: TCP port: 80 targetPort: 8080 ``` In this example, the Service named my-load-balancer will balance traffic to the Pods labeled with app: my-web-app. Adjust the ports as needed. Create a YAML file for the Pod on the first EC2 instance. Let's name it instance-1.yaml: ``` apiVersion: v1 kind: Pod metadata: name: instance-1 labels: app: my-web-app spec: containers: - name: web-server image: my-web-server-image:latest ports: - containerPort: 8080 ``` Replace instance-1 with a suitable name and adjust the container image and port based on your web server setup. Create a YAML file for the Pod on the second EC2 instance. Let's name it instance-2.yaml: ``` apiVersion: v1 kind: Pod metadata: name: instance-2 labels: app: my-web-app spec: containers: - name: web-server image: my-web-server-image:latest ports: - containerPort: 8080 ``` Replace instance-2 with a suitable name and adjust the container image and port as needed. Apply the YAML files to create the LoadBalancer Service and the Pods: ``` kubectl apply -f loadbalancer.yaml kubectl apply -f instance-1.yaml kubectl apply -f instance-2.yaml ``` Once applied, the LoadBalancer Service will distribute traffic to the Pods running on the two EC2 instances. ## memory monitor tool install ps_mem tool ```sudo pip install ps_mem``` add \$HOME/.local/bin to PATH by modify ~/.bashrc ``vi ~/.bashrc`` add the following line in the bottom of ~/.bashrc ``export PATH="$PATH:$HOME/.local/bin"`` apply the ~/.bashrc again ``source ~/.bashrc`` now you can use `sudo ps_mem` to get a detail memory usage info. ``` Private + Shared = RAM used Program 148.0 KiB + 16.5 KiB = 164.5 KiB gsd-xsettings 36.6 MiB + 11.5 KiB = 36.6 MiB Xwayland 70.7 MiB + 156.5 KiB = 70.9 MiB dockerd 122.9 MiB + 31.1 MiB = 154.0 MiB node (4) 149.9 MiB + 26.0 MiB = 175.8 MiB gnome-shell --------------------------------- 1.0 GiB ================================= ``` 1. delete etcd sudo rm -rf /var/lib/etcd/member 2. backup data sudo cp -r /data/k8s-pv /data/k8s-pv-backup 3. reset k8s sudo kubeadm reset --cri-socket=unix:///var/run/cri-dockerd.sock and got this ``` ubuntu@ddbrothers:~$ sudo kubeadm reset --cri-socket=unix:///var/run/cri-dockerd.sock [reset] Reading configuration from the cluster... [reset] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' W0520 21:58:15.625280 65850 reset.go:106] [reset] Unable to fetch the kubeadm-config ConfigMap from cluster: failed to get config map: Get "https://192.168.1.221:6443/api/v1/namespaces/kube-system/configmaps/kubeadm-config?timeout=10s": dial tcp 192.168.1.221:6443: connect: connection refused W0520 21:58:15.625390 65850 preflight.go:56] [reset] WARNING: Changes made to this host by 'kubeadm init' or 'kubeadm join' will be reverted. [reset] Are you sure you want to proceed? [y/N]: y [preflight] Running pre-flight checks W0520 21:58:30.641167 65850 removeetcdmember.go:106] [reset] No kubeadm config, using etcd pod spec to get data directory [reset] Stopping the kubelet service [reset] Unmounting mounted directories in "/var/lib/kubelet" [reset] Deleting contents of directories: [/etc/kubernetes/manifests /var/lib/kubelet /etc/kubernetes/pki] [reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf] The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d The reset process does not reset or clean up iptables rules or IPVS tables. If you wish to reset iptables, you must do so manually by using the "iptables" command. If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar) to reset your system's IPVS tables. The reset process does not clean your kubeconfig files and you must remove them manually. Please, check the contents of the $HOME/.kube/config file. ubuntu@ddbrothers:~$ ```