---
# System prepended metadata

title: Step by step install HA Kubernetes with Kubeadm in Disconnected environment

---

# Step by step install HA Kubernetes with Kubeadm in Disconnected environment

## 0. Preface

本篇文章會介紹，如何在全離線的環境下，在多台 VM 上使用 `kubeadm` 安裝 3 台 Control-plane Node 和 3 台 Worker Node 架構的 Kubernetes Cluster

### 0.1. 環境架構規劃

![image](https://hackmd.io/_uploads/HJVAgQ0aex.png)

## 1. 準備 K8s 全離線安裝包

### 1.1. 準備 K8s 軟體套件

1. 連線到一台 Disk 空間 70G 或以上, 可以透過網路正常連線到公有網路的 RHEL 9 電腦
2. 註冊 RHEL 9

    ```
    sudo rhc connect --activation-key $ACTIVATION-KEY --organization $ORGANIZATION
    ```
3. 設定 Kubernetes 和 CRI-O 版本

    ```
    KUBERNETES_VERSION=v1.34
    CRIO_VERSION=v1.34
    ```
4. 添加 Kubernetes yum repository

    ```
    cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
    [kubernetes]
    name=Kubernetes
    baseurl=https://pkgs.k8s.io/core:/stable:/$KUBERNETES_VERSION/rpm/
    enabled=1
    gpgcheck=1
    gpgkey=https://pkgs.k8s.io/core:/stable:/$KUBERNETES_VERSION/rpm/repodata/repomd.xml.key
    EOF
    ```

5. 添加 CRI-O yum repository

    ```
    cat <<EOF | sudo tee /etc/yum.repos.d/cri-o.repo
    [cri-o]
    name=CRI-O
    baseurl=https://download.opensuse.org/repositories/isv:/cri-o:/stable:/$CRIO_VERSION/rpm/
    enabled=1
    gpgcheck=1
    gpgkey=https://download.opensuse.org/repositories/isv:/cri-o:/stable:/$CRIO_VERSION/rpm/repodata/repomd.xml.key
    EOF
    ```

6. 建立工作目錄

    ```
    mkdir -p "$HOME"/k8s/{addon/{cni/cilium,csi/nfs-provisioner,ingress/ingress-nginx,metrics/metrics-server,logging/{fluentd,logging-operator},kube-vip/{ha-control-plane,k8s-lb-service}},img,pkg/{kubernetes,podman,skopeo},config}
    ```

7. 目錄結構如下

    ```
    tree "$HOME"/k8s
    ```

    執行結果：

    ```
    /home/bigred/k8s
    ├── addon
    │   ├── cni
    │   │   └── cilium
    │   ├── csi
    │   │   └── nfs-provisioner
    │   ├── ingress
    │   │   └── ingress-nginx
    │   ├── kube-vip
    │   │   ├── ha-control-plane
    │   │   └── k8s-lb-service
    │   ├── logging
    │   │   ├── fluentd
    │   │   └── logging-operator
    │   └── metrics
    │       └── metrics-server
    ├── config
    ├── img
    └── pkg
        ├── kubernetes
        ├── podman
        └── skopeo

    21 directories, 0 files
    ```


8. 將必要套件載入指定目錄

    ```
    sudo yum install --downloadonly --downloaddir="$HOME"/k8s/pkg/kubernetes -y \
      crun \
      runc \
      cri-o \
      kubeadm \
      kubectl \
      kubelet
    ```

9. 檢視已載入的套件

    ```
    ls -lh "$HOME"/k8s/pkg/kubernetes
    ```

    執行結果：

    ```
    total 78M
    -rw-r--r--. 1 root root 240K Oct 16 16:28 conntrack-tools-1.4.7-4.el9_5.x86_64.rpm
    -rw-r--r--. 1 root root  66K Oct 16 16:28 container-selinux-2.235.0-1.el9_6.noarch.rpm
    -rw-r--r--. 1 root root  21M Oct 16 16:28 cri-o-1.34.1-150500.1.1.x86_64.rpm
    -rw-r--r--. 1 root root 7.3M Oct 16 16:28 cri-tools-1.34.0-150500.1.1.x86_64.rpm
    -rw-r--r--. 1 root root 563K Oct 16 16:28 criu-3.19-1.el9.x86_64.rpm
    -rw-r--r--. 1 root root  33K Oct 16 16:28 criu-libs-3.19-1.el9.x86_64.rpm
    -rw-r--r--. 1 root root 244K Oct 16 16:28 crun-1.23.1-2.el9_6.x86_64.rpm
    -rw-r--r--. 1 root root  13M Oct 16 16:28 kubeadm-1.34.1-150500.1.1.x86_64.rpm
    -rw-r--r--. 1 root root  12M Oct 16 16:28 kubectl-1.34.1-150500.1.1.x86_64.rpm
    -rw-r--r--. 1 root root  13M Oct 16 16:28 kubelet-1.34.1-150500.1.1.x86_64.rpm
    -rw-r--r--. 1 root root 8.6M Oct 16 16:28 kubernetes-cni-1.7.1-150500.1.1.x86_64.rpm
    -rw-r--r--. 1 root root  60K Oct 16 16:28 libnet-1.2-7.el9.x86_64.rpm
    -rw-r--r--. 1 root root  26K Oct 16 16:28 libnetfilter_cthelper-1.0.0-22.el9.x86_64.rpm
    -rw-r--r--. 1 root root  26K Oct 16 16:28 libnetfilter_cttimeout-1.0.0-19.el9.x86_64.rpm
    -rw-r--r--. 1 root root  31K Oct 16 16:28 libnetfilter_queue-1.0.5-1.el9.x86_64.rpm
    -rw-r--r--. 1 root root  38K Oct 16 16:28 protobuf-c-1.3.3-13.el9.x86_64.rpm
    -rw-r--r--. 1 root root 3.6M Oct 16 16:28 runc-1.2.4-2.el9.x86_64.rpm
    -rw-r--r--. 1 root root  42K Oct 16 16:28 yajl-2.1.0-25.el9.x86_64.rpm
    ```

10. 下載並準備 podman pkg

    ```
    sudo yum install --downloadonly --downloaddir="$HOME"/k8s/pkg/podman -y \
      podman
    ```

11. 檢視已載入的套件

    ```
    ls -l "$HOME"/k8s/pkg/podman
    ```

    執行結果：

    ```
    total 24048
    -rw-r--r--. 1 bigred bigred   902510 Oct 16 16:59  aardvark-dns-1.14.0-1.el9.x86_64.rpm
    -rw-r--r--. 1 bigred bigred    55519 Oct 16 16:59  conmon-2.1.12-1.el9.x86_64.rpm
    -rw-r--r--. 1 bigred bigred   158417 Oct 16 16:59  containers-common-1-117.el9_6.x86_64.rpm
    -rw-r--r--. 1 bigred bigred    67105 Oct 16 16:59  container-selinux-2.235.0-1.el9_6.noarch.rpm
    -rw-r--r--. 1 bigred bigred   576009 Oct 16 16:59  criu-3.19-1.el9.x86_64.rpm
    -rw-r--r--. 1 bigred bigred    33643 Oct 16 16:59  criu-libs-3.19-1.el9.x86_64.rpm
    -rw-r--r--. 1 bigred bigred   249255 Oct 16 16:59  crun-1.23.1-2.el9_6.x86_64.rpm
    -rw-r--r--. 1 bigred bigred    58706 Oct 16 16:59  fuse3-3.10.2-9.el9.x86_64.rpm
    -rw-r--r--. 1 bigred bigred    95573 Oct 16 16:59  fuse3-libs-3.10.2-9.el9.x86_64.rpm
    -rw-r--r--. 1 bigred bigred     8750 Oct 16 16:59  fuse-common-3.10.2-9.el9.x86_64.rpm
    -rw-r--r--. 1 bigred bigred    71022 Oct 16 16:59  fuse-overlayfs-1.14-1.el9.x86_64.rpm
    -rw-r--r--. 1 bigred bigred    61278 Oct 16 16:59  libnet-1.2-7.el9.x86_64.rpm
    -rw-r--r--. 1 bigred bigred    71992 Oct 16 16:59  libslirp-4.4.0-8.el9.x86_64.rpm
    -rw-r--r--. 1 bigred bigred  3846970 Oct 16 16:59  netavark-1.14.1-1.el9_6.x86_64.rpm
    -rw-r--r--. 1 bigred bigred   268131 Oct 16 16:59 'passt-0^20250217.ga1e48a0-10.el9_6.x86_64.rpm'
    -rw-r--r--. 1 bigred bigred    27658 Oct 16 16:59 'passt-selinux-0^20250217.ga1e48a0-10.el9_6.noarch.rpm'
    -rw-r--r--. 1 bigred bigred 17803252 Oct 16 16:59  podman-5.4.0-13.el9_6.x86_64.rpm
    -rw-r--r--. 1 bigred bigred    38224 Oct 16 16:59  protobuf-c-1.3.3-13.el9.x86_64.rpm
    -rw-r--r--. 1 bigred bigred    90389 Oct 16 16:59  shadow-utils-subid-4.9-12.el9.x86_64.rpm
    -rw-r--r--. 1 bigred bigred    50387 Oct 16 16:59  slirp4netns-1.3.2-1.el9.x86_64.rpm
    -rw-r--r--. 1 bigred bigred    42487 Oct 16 16:59  yajl-2.1.0-25.el9.x86_64.rpm
    ```

12. 下載並準備 skopeo pkg

    ```
    sudo yum install --downloadonly --downloaddir="$HOME"/k8s/pkg/skopeo -y \
      skopeo
    ```

13. 檢視已載入的套件

    ```
    ls -l "$HOME"/k8s/pkg/skopeo
    ```

    執行結果：

    ```
    total 9308
    -rw-r--r--. 1 root root 9530108 Oct 17 10:19 skopeo-1.18.1-2.el9_6.x86_64.rpm
    ```

### 1.2. 準備 kubeadm 要安裝 K8s 所需的 images

1. 列出安裝 kubeadm 要安裝 K8s 所需的 images

    ```
    kubeadm config images list | tee ${HOME}/k8s/img/img-list.txt
    ```

    執行結果：

    ```
    registry.k8s.io/kube-apiserver:v1.34.1
    registry.k8s.io/kube-controller-manager:v1.34.1
    registry.k8s.io/kube-scheduler:v1.34.1
    registry.k8s.io/kube-proxy:v1.34.1
    registry.k8s.io/coredns/coredns:v1.12.1
    registry.k8s.io/pause:3.10.1
    registry.k8s.io/etcd:3.6.4-0
    ```

2. 將 K8s images 備份到指定目錄中

    ```
    for i in $(cat ${HOME}/k8s/img/img-list.txt); do podman pull ${i}; done
    podman save -m $(cat ${HOME}/k8s/img/img-list.txt) > ${HOME}/k8s/img/all-K8s-Components-v1.34.1.tar
    ```

3. 拷貝 K8s 所有 components 的 images 到 Harbor 的 library project 中

    ```
    for i in $(cat ${HOME}/k8s/img/img-list.txt); do skopeo copy docker://${i} docker://${i/registry.k8s.io/${HARBOR_FQDN}/library}; done
    ```


#### 1.2.3. 準備 ingress-nginx `v1.13.3` 所需之 images 和安裝 yaml

1. 找到 ingress-nginx `v1.13.3` 所需之 images

    ```
    INGRESS_NGINX_VERSION="v1.13.3"
    curl -s https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-${INGRESS_NGINX_VERSION}/deploy/static/provider/cloud/deploy.yaml | grep image: | sort -u | tr -s " " | cut -d " " -f 3 | tee -a ${HOME}/k8s/img/img-list.txt
    ```

    執行結果：

    ```
    registry.k8s.io/ingress-nginx/controller:v1.13.3@sha256:1b044f6dcac3afbb59e05d98463f1dec6f3d3fb99940bc12ca5d80270358e3bd
    registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.6.3@sha256:3d671cf20a35cd94efc5dcd484970779eb21e7938c98fbc3673693b8a117cf39
    ```

2. 將 Ingress-nginx Controller image 備份到指定目錄

    ```
    for i in $(grep ingress-nginx ${HOME}/k8s/img/img-list.txt); do podman pull ${i}; podman tag ${i} ${i%@*};done
    podman save -m $(grep ingress-nginx ${HOME}/k8s/img/img-list.txt | cut -d "@" -f 1) > ${HOME}/k8s/img/ingress-nginx-controller-v1.13.3.tar
    ```

3. 將安裝 yaml 下載至指定目錄

    ```
    curl -so ${HOME}/k8s/addon/ingress/ingress-nginx/deploy.yaml https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-${INGRESS_NGINX_VERSION}/deploy/static/provider/cloud/deploy.yaml
    ```

### 1.3. 準備 kube-vip `v1.0.1` 所需之 images

1. 設定環境資訊

    ```
    export VIP="172.20.6.110"
    export VIPSUBSET="16"
    export INTERFACE="ens18"
    KVVERSION="v1.0.1"
    
    alias kube-vip="sudo podman run --network host --rm ghcr.io/kube-vip/kube-vip:$KVVERSION"
    ```

2. 產生 kube-vip static pod yaml

    ```
    kube-vip manifest pod \
        --interface $INTERFACE \
        --address $VIP \
        --vipSubnet $VIPSUBSET \
        --controlplane \
        --arp \
        --leaderElection | tee ${HOME}/k8s/addon/kube-vip/ha-control-plane/kube-vip-static-pod.yaml
    ````
    > 上述參數沒有 `--service` 是因為 control-plane 的 static pod 不需要管理 loadbalancer type service

3. 產生 kube-vip daemonset yaml

    ```
    kube-vip manifest daemonset \
        --interface $INTERFACE \
        --vipSubnet $VIPSUBSET \
        --services \
        --arp \
        --leaderElection \
        --servicesElection | tee ${HOME}/k8s/addon/kube-vip/k8s-lb-service/kube-vip-ds.yaml
    ```

4. 下載 kube-vip Cloud Provider yaml 檔

    ```
    curl -so ${HOME}/k8s/addon/kube-vip/k8s-lb-service/kube-vip-cloud-controller.yaml https://raw.githubusercontent.com/kube-vip/kube-vip-cloud-provider/main/manifest/kube-vip-cloud-controller.yaml
    ```

5. 下載 kube-vip daemonset 所需之 rbac.yaml

    ```
    curl -so ${HOME}/k8s/addon/kube-vip/k8s-lb-service/kube-vip-rbac.yaml https://kube-vip.io/manifests/rbac.yaml
    ```

5. 檢視 Kubevip 執行所需之 images

    ```
    find ${HOME}/k8s/addon/kube-vip -name "*.yaml" -type f -exec cat {} \; | grep image: | tr -s " " | uniq | cut -d " " -f 3 | tee -a ${HOME}/k8s/img/img-list.txt
    ```

    執行結果：

    ```
    ghcr.io/kube-vip/kube-vip:v1.0.1
    ghcr.io/kube-vip/kube-vip-cloud-provider:v0.0.12
    ```

6. 將 kube-vip images 備份到指定目錄

    ```
    for i in $(tail -n 2 ${HOME}/k8s/img/img-list.txt); do podman pull ${i}; done
    podman save -m $(tail -n 2 ${HOME}/k8s/img/img-list.txt) > ${HOME}/k8s/img/kube-vip-v1.0.1.tar
    ```

### 1.4. 準備 Metrics Server `v0.8.0` 所需之 images

1. 定義 Metrics Server 版本變數

    ```
    METRICS_SERVER_VERSION="v0.8.0"
    ```

2. 下載安裝 yaml 到指定路徑

    ```
    curl -sLo ${HOME}/k8s/addon/metrics/metrics-server/components.yaml https://github.com/kubernetes-sigs/metrics-server/releases/download/${METRICS_SERVER_VERSION}/components.yaml
    ```

3. 檢視 Metrics Server 執行所需之 images

    ```
    find ${HOME}/k8s/addon/metrics/metrics-server -name "*.yaml" -type f -exec cat {} \; | grep image: | tr -s " " | uniq | cut -d " " -f 3 | tee -a ${HOME}/k8s/img/img-list.txt
    ```

    執行結果：

    ```
    registry.k8s.io/metrics-server/metrics-server:v0.8.0
    ```

4. 將 Metrics Server image 備份到指定目錄

    ```
    for i in $(grep metrics-server ${HOME}/k8s/img/img-list.txt); do podman pull ${i}; done
    podman save -m $(grep metrics-server ${HOME}/k8s/img/img-list.txt) > ${HOME}/k8s/img/metrics-server-${METRICS_SERVER_VERSION}.tar
    ```

### 1.5. 準備 Fluentd `v1.19.0` 所需之 images

1. 定義 Fluentd 版本變數

    ```
    FLUENTED_VERSION="v1.19.0-1.1"
    ```

2. 下載安裝 yaml 到指定路徑

    ```
    curl -sLo ${HOME}/k8s/addon/logging/fluentd/fluentd-daemonset-syslog.yaml https://raw.githubusercontent.com/fluent/fluentd-kubernetes-daemonset/refs/tags/${FLUENTED_VERSION}/fluentd-daemonset-syslog.yaml
    ```

3. 修改 yaml

    ```
    sed -i "s|version: v1|version: ${FLUENTED_VERSION}|g" fluentd-daemonset-syslog.yaml
    sed -i "s|v1-debian-syslog|v1.19.0-debian-syslog-amd64-1.1|g" fluentd-daemonset-syslog.yaml
    sed -i 's|fluent/fluentd-kubernetes-daemonset|docker.io/fluent/fluentd-kubernetes-daemonset|g' fluentd-daemonset-syslog.yaml
    ```

3. 檢視 fluentd 執行所需之 images

    ```
    find ${HOME}/k8s/addon/logging/fluentd -name "*.yaml" -type f -exec cat {} \; | grep image: | tr -s " " | uniq | cut -d " " -f 3 | tee -a ${HOME}/k8s/img/img-list.txt
    ```

    執行結果：

    ```
    docker.io/fluent/fluentd-kubernetes-daemonset:v1.19.0-debian-syslog-amd64-1.1
    ```

4. 將 fluentd image 備份到指定目錄

    ```
    for i in $(grep fluentd ${HOME}/k8s/img/img-list.txt); do podman pull ${i}; done
    podman save -m $(grep fluentd ${HOME}/k8s/img/img-list.txt) > ${HOME}/k8s/img/fluentd-kubernetes-daemonset-v1.19.0-debian-syslog-amd64-1.1.tar
    ```

### 1.6. 準備 cilium `1.18.2` 所需之 images

1. 定義 helm 和 cilium 版本變數

    ```
    HELM_VERSION="v3.19.0"
    CILIUM_VERSION="1.18.2"
    ```

2. 下載 helm Binary

    ```
    curl -sLo ${HOME}/k8s/addon/cni/cilium/helm-${HELM_VERSION}-linux-amd64.tar.gz https://get.helm.sh/helm-${HELM_VERSION}-linux-amd64.tar.gz
    ```

3. 添加 Cilium Helm repository

    ```
    helm repo add cilium https://helm.cilium.io/
    ```


2. 下載 cilium chart file

    ```
    helm fetch cilium/cilium --version=${CILIUM_VERSION} -d ${HOME}/k8s/addon/cni/cilium/
    ```

3. 設定 helm values yaml

    ```
    echo 'bpf:
      masquerade: true
    hubble:
      relay:
        enabled: true
      ui:
        enabled: true
        service:
          type: NodePort
    ipam:
      mode: kubernetes
    ipv4NativeRoutingCIDR: 10.244.0.0/16
    k8sServiceHost: 172.20.6.10
    k8sServicePort: 6443
    kubeProxyReplacement: true' > ${HOME}/k8s/addon/cni/cilium/custom-values.yaml
    ```

3. 檢視 cilium 執行所需之 images

    ```
    helm template ${HOME}/k8s/addon/cni/cilium/cilium-${CILIUM_VERSION}.tgz -f ${HOME}/k8s/addon/cni/cilium/custom-values.yaml | awk '$1 ~ /image:/ {print $2}' | sed -e 's/\"//g' | sort -u | tee -a ${HOME}/k8s/img/img-list.txt
    ```

    執行結果：

    ```
    quay.io/cilium/cilium-envoy:v1.34.7-1757592137-1a52bb680a956879722f48c591a2ca90f7791324@sha256:7932d656b63f6f866b6732099d33355184322123cfe1182e6f05175a3bc2e0e0
    quay.io/cilium/cilium:v1.18.2@sha256:858f807ea4e20e85e3ea3240a762e1f4b29f1cb5bbd0463b8aa77e7b097c0667
    quay.io/cilium/hubble-relay:v1.18.2@sha256:6079308ee15e44dff476fb522612732f7c5c4407a1017bc3470916242b0405ac
    quay.io/cilium/hubble-ui-backend:v0.13.3@sha256:db1454e45dc39ca41fbf7cad31eec95d99e5b9949c39daaad0fa81ef29d56953
    quay.io/cilium/hubble-ui:v0.13.3@sha256:661d5de7050182d495c6497ff0b007a7a1e379648e60830dd68c4d78ae21761d
    quay.io/cilium/operator-generic:v1.18.2@sha256:cb4e4ffc5789fd5ff6a534e3b1460623df61cba00f5ea1c7b40153b5efb81805
    ```

4. 將 cilium images 備份到指定目錄

    ```
    for i in $(grep "quay.io/cilium" ${HOME}/k8s/img/img-list.txt); do podman pull ${i}; podman tag ${i} ${i%@*}; done
    podman save -m $(grep "quay.io/cilium" ${HOME}/k8s/img/img-list.txt | cut -d "@" -f 1) > ${HOME}/k8s/img/cilium-${CILIUM_VERSION}.tar
    ```

5. 下載 Cilium CLI

    ```
    CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
    CLI_ARCH=amd64
    if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
    curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
    sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
    ```

6. 下載 Hubble Client

    ```
    HUBBLE_VERSION="$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)"
    HUBBLE_ARCH=amd64
    if [ "$(uname -m)" = "aarch64" ]; then HUBBLE_ARCH=arm64; fi
    curl -L --fail --remote-name-all https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-${HUBBLE_ARCH}.tar.gz{,.sha256sum}
    sha256sum --check hubble-linux-${HUBBLE_ARCH}.tar.gz.sha256sum
    ```

10. 準備 cilium 測試網路效能 image

    ```
    sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
    cilium connectivity perf --print-image-artifacts | tee -a ${HOME}/k8s/img/img-list.txt
    ```

    執行結果：

    ```
    quay.io/cilium/network-perf:1751527436-c2462ae@sha256:0c491ed7ca63e6c526593b3a2d478f856410a50fbbce7fe2b64283c3015d752f
    ```

11. 將 cilium 測試網路效能 images 備份到指定目錄

    ```
    for i in $(grep "network-perf" ${HOME}/k8s/img/img-list.txt); do podman pull ${i}; podman tag ${i} ${i%@*}; done
    podman save $(grep "network-perf" ${HOME}/k8s/img/img-list.txt | cut -d "@" -f 1) > ${HOME}/k8s/img/cilium-network-perf.tar
    ```

### 1.7. 準備 NFS subdir external provisioner `v4.0.2` 所需之 images

1. 定義 NFS provisioner 版本變數

    ```
    NFS_PROVISIONER_VERSION="v4.0.2"
    ```

2. 下載安裝 yaml 到指定路徑

    ```
    git clone --no-checkout --depth=1 --filter=tree:0 https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner.git ${HOME}/k8s/addon/csi/nfs-provisioner/nfs-subdir-external-provisioner
    cd ${HOME}/k8s/addon/csi/nfs-provisioner/nfs-subdir-external-provisioner
    git sparse-checkout set --no-cone /deploy
    git checkout
    mv ${HOME}/k8s/addon/csi/nfs-provisioner/nfs-subdir-external-provisioner/deploy/ ${HOME}/k8s/addon/csi/nfs-provisioner/
    sudo rm -r ${HOME}/k8s/addon/csi/nfs-provisioner/nfs-subdir-external-provisioner
    sed -i "s|busybox:stable|docker.io/library/busybox:stable|g" ${HOME}/k8s/addon/csi/nfs-provisioner/deploy/test-pod.yaml
    ```

3. 建立 `kustomization.yaml`

    ```
    echo 'namespace: nfs-provisioner
    bases:
      - ./deploy
    resources:
      - namespace.yaml
    patchesStrategicMerge:
      - patch_nfs_details.yaml' > ${HOME}/k8s/addon/csi/nfs-provisioner/kustomization.yaml
    ```

4. 建立 `namespace.yaml`

    ```
    echo '# namespace.yaml
    apiVersion: v1
    kind: Namespace
    metadata:
      name: nfs-provisioner' > ${HOME}/k8s/addon/csi/nfs-provisioner/namespace.yaml
    ```

5. 建立 `patch_nfs_details.yaml`

    ```
    echo 'apiVersion: apps/v1
    kind: Deployment
    metadata:
      labels:
        app: nfs-client-provisioner
      name: nfs-client-provisioner
    spec:
      template:
        spec:
          containers:
            - name: nfs-client-provisioner
              env:
                - name: NFS_SERVER
                  value: 172.20.6.9
                - name: NFS_PATH
                  value: /k8s
          volumes:
            - name: nfs-client-root
              nfs:
                server: 172.20.6.9
                path: /k8s' > ${HOME}/k8s/addon/csi/nfs-provisioner/patch_nfs_details.yaml
    ```

6. 檢視 nfs-provisioner 執行所需之 images

    ```
    find ${HOME}/k8s/addon/csi/nfs-provisioner -name "*.yaml" -type f -exec cat {} \; | grep image: | tr -s " " | uniq | cut -d " " -f 3 | tee -a ${HOME}/k8s/img/img-list.txt
    ```

    執行結果：

    ```
    registry.k8s.io/sig-storage/nfs-subdir-external-provisioner:v4.0.2
    docker.io/library/busybox:stable
    ```

4. 將 nfs-provisioner images 備份到指定目錄

    ```
    for i in $(grep -E "nfs-subdir-external-provisioner|busybox" ${HOME}/k8s/img/img-list.txt); do podman pull ${i}; done
    podman save -m $(grep -E "nfs-subdir-external-provisioner|busybox" ${HOME}/k8s/img/img-list.txt) > ${HOME}/k8s/img/nfs-provisioner-${NFS_PROVISIONER_VERSION}.tar
    ```

### 1.8. 準備 Ingress NGINX `v1.13.3` 所需之 images 和 yaml 檔

1. 定義 Ingress NGINX 版本變數

    ```
    INGRESS_NGINX="v1.13.3"
    ```

2. 下載部署 Ingress NGINX YAML

    ```
    curl -sLo ${HOME}/k8s/addon/ingress/ingress-nginx/deployment.yaml \
    https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-${INGRESS_NGINX}/deploy/static/provider/baremetal/deploy.yaml
    ```

3. 將 deployment 轉成 daemonset

    ```
    nano daemonSet.yaml
    ```

    :::spoiler 檔案內容：

    ```
    apiVersion: v1
    kind: Namespace
    metadata:
      labels:
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
      name: ingress-nginx
    ---
    apiVersion: v1
    automountServiceAccountToken: true
    kind: ServiceAccount
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.13.3
      name: ingress-nginx
      namespace: ingress-nginx
    ---
    apiVersion: v1
    automountServiceAccountToken: true
    kind: ServiceAccount
    metadata:
      labels:
        app.kubernetes.io/component: admission-webhook
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.13.3
      name: ingress-nginx-admission
      namespace: ingress-nginx
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.13.3
      name: ingress-nginx
      namespace: ingress-nginx
    rules:
    - apiGroups:
      - ""
      resources:
      - namespaces
      verbs:
      - get
    - apiGroups:
      - ""
      resources:
      - configmaps
      - pods
      - secrets
      - endpoints
      verbs:
      - get
      - list
      - watch
    - apiGroups:
      - ""
      resources:
      - services
      verbs:
      - get
      - list
      - watch
    - apiGroups:
      - networking.k8s.io
      resources:
      - ingresses
      verbs:
      - get
      - list
      - watch
    - apiGroups:
      - networking.k8s.io
      resources:
      - ingresses/status
      verbs:
      - update
    - apiGroups:
      - networking.k8s.io
      resources:
      - ingressclasses
      verbs:
      - get
      - list
      - watch
    - apiGroups:
      - coordination.k8s.io
      resourceNames:
      - ingress-nginx-leader
      resources:
      - leases
      verbs:
      - get
      - update
    - apiGroups:
      - coordination.k8s.io
      resources:
      - leases
      verbs:
      - create
    - apiGroups:
      - ""
      resources:
      - events
      verbs:
      - create
      - patch
    - apiGroups:
      - discovery.k8s.io
      resources:
      - endpointslices
      verbs:
      - list
      - watch
      - get
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      labels:
        app.kubernetes.io/component: admission-webhook
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.13.3
      name: ingress-nginx-admission
      namespace: ingress-nginx
    rules:
    - apiGroups:
      - ""
      resources:
      - secrets
      verbs:
      - get
      - create
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      labels:
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.13.3
      name: ingress-nginx
    rules:
    - apiGroups:
      - ""
      resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
      - namespaces
      verbs:
      - list
      - watch
    - apiGroups:
      - coordination.k8s.io
      resources:
      - leases
      verbs:
      - list
      - watch
    - apiGroups:
      - ""
      resources:
      - nodes
      verbs:
      - get
    - apiGroups:
      - ""
      resources:
      - services
      verbs:
      - get
      - list
      - watch
    - apiGroups:
      - networking.k8s.io
      resources:
      - ingresses
      verbs:
      - get
      - list
      - watch
    - apiGroups:
      - ""
      resources:
      - events
      verbs:
      - create
      - patch
    - apiGroups:
      - networking.k8s.io
      resources:
      - ingresses/status
      verbs:
      - update
    - apiGroups:
      - networking.k8s.io
      resources:
      - ingressclasses
      verbs:
      - get
      - list
      - watch
    - apiGroups:
      - discovery.k8s.io
      resources:
      - endpointslices
      verbs:
      - list
      - watch
      - get
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      labels:
        app.kubernetes.io/component: admission-webhook
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.13.3
      name: ingress-nginx-admission
    rules:
    - apiGroups:
      - admissionregistration.k8s.io
      resources:
      - validatingwebhookconfigurations
      verbs:
      - get
      - update
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.13.3
      name: ingress-nginx
      namespace: ingress-nginx
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: Role
      name: ingress-nginx
    subjects:
    - kind: ServiceAccount
      name: ingress-nginx
      namespace: ingress-nginx
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      labels:
        app.kubernetes.io/component: admission-webhook
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.13.3
      name: ingress-nginx-admission
      namespace: ingress-nginx
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: Role
      name: ingress-nginx-admission
    subjects:
    - kind: ServiceAccount
      name: ingress-nginx-admission
      namespace: ingress-nginx
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      labels:
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.13.3
      name: ingress-nginx
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: ingress-nginx
    subjects:
    - kind: ServiceAccount
      name: ingress-nginx
      namespace: ingress-nginx
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      labels:
        app.kubernetes.io/component: admission-webhook
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.13.3
      name: ingress-nginx-admission
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: ingress-nginx-admission
    subjects:
    - kind: ServiceAccount
      name: ingress-nginx-admission
      namespace: ingress-nginx
    ---
    apiVersion: v1
    data: null
    kind: ConfigMap
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.13.3
      name: ingress-nginx-controller
      namespace: ingress-nginx
    ---
    apiVersion: v1
    kind: Service
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.13.3
      name: ingress-nginx-controller
      namespace: ingress-nginx
    spec:
      externalTrafficPolicy: Local
      ipFamilies:
      - IPv4
      ipFamilyPolicy: SingleStack
      ports:
      - appProtocol: http
        name: http
        port: 80
        protocol: TCP
        targetPort: http
      - appProtocol: https
        name: https
        port: 443
        protocol: TCP
        targetPort: https
      selector:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
      type: LoadBalancer
    ---
    apiVersion: v1
    kind: Service
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.13.3
      name: ingress-nginx-controller-admission
      namespace: ingress-nginx
    spec:
      ports:
      - appProtocol: https
        name: https-webhook
        port: 443
        targetPort: webhook
      selector:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
      type: ClusterIP
    ---
    apiVersion: apps/v1
    kind: DaemonSet
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.13.3
      name: ingress-nginx-controller
      namespace: ingress-nginx
    spec:
      selector:
        matchLabels:
          app.kubernetes.io/component: controller
          app.kubernetes.io/instance: ingress-nginx
          app.kubernetes.io/name: ingress-nginx
      template:
        metadata:
          labels:
            app.kubernetes.io/component: controller
            app.kubernetes.io/instance: ingress-nginx
            app.kubernetes.io/name: ingress-nginx
            app.kubernetes.io/part-of: ingress-nginx
            app.kubernetes.io/version: 1.13.3
        spec:
          automountServiceAccountToken: true
          containers:
          - args:
            - /nginx-ingress-controller
            - --election-id=ingress-nginx-leader
            - --controller-class=k8s.io/ingress-nginx
            - --ingress-class=nginx
            - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
            - --validating-webhook=:8443
            - --validating-webhook-certificate=/usr/local/certificates/cert
            - --validating-webhook-key=/usr/local/certificates/key
            - --enable-metrics=false
            - --report-node-internal-ip-address=true
            - --watch-ingress-without-class=true
            env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: LD_PRELOAD
              value: /usr/local/lib/libmimalloc.so
            image: registry.k8s.io/ingress-nginx/controller:v1.13.3@sha256:1b044f6dcac3afbb59e05d98463f1dec6f3d3fb99940bc12ca5d80270358e3bd
            imagePullPolicy: IfNotPresent
            lifecycle:
              preStop:
                exec:
                  command:
                  - /wait-shutdown
            livenessProbe:
              failureThreshold: 5
              httpGet:
                path: /healthz
                port: 10254
                scheme: HTTP
              initialDelaySeconds: 10
              periodSeconds: 10
              successThreshold: 1
              timeoutSeconds: 1
            name: controller
            ports:
            - containerPort: 80
              name: http
              protocol: TCP
              hostPort: 80
            - containerPort: 443
              name: https
              protocol: TCP
              hostPort: 443
            - containerPort: 8443
              name: webhook
              protocol: TCP
            readinessProbe:
              failureThreshold: 3
              httpGet:
                path: /healthz
                port: 10254
                scheme: HTTP
              initialDelaySeconds: 10
              periodSeconds: 10
              successThreshold: 1
              timeoutSeconds: 1
            resources:
              requests:
                cpu: 100m
                memory: 90Mi
            securityContext:
              allowPrivilegeEscalation: false
              capabilities:
                add:
                - NET_BIND_SERVICE
                drop:
                - ALL
              readOnlyRootFilesystem: false
              runAsGroup: 82
              runAsNonRoot: true
              runAsUser: 101
              seccompProfile:
                type: RuntimeDefault
            volumeMounts:
            - mountPath: /usr/local/certificates/
              name: webhook-cert
              readOnly: true
          dnsPolicy: ClusterFirst
          nodeSelector:
            kubernetes.io/os: linux
          serviceAccountName: ingress-nginx
          terminationGracePeriodSeconds: 300
          volumes:
          - name: webhook-cert
            secret:
              secretName: ingress-nginx-admission
    ---
    apiVersion: batch/v1
    kind: Job
    metadata:
      labels:
        app.kubernetes.io/component: admission-webhook
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.13.3
      name: ingress-nginx-admission-create
      namespace: ingress-nginx
    spec:
      template:
        metadata:
          labels:
            app.kubernetes.io/component: admission-webhook
            app.kubernetes.io/instance: ingress-nginx
            app.kubernetes.io/name: ingress-nginx
            app.kubernetes.io/part-of: ingress-nginx
            app.kubernetes.io/version: 1.13.3
          name: ingress-nginx-admission-create
        spec:
          automountServiceAccountToken: true
          containers:
          - args:
            - create
            - --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc
            - --namespace=$(POD_NAMESPACE)
            - --secret-name=ingress-nginx-admission
            env:
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            image: registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.6.3@sha256:3d671cf20a35cd94efc5dcd484970779eb21e7938c98fbc3673693b8a117cf39
            imagePullPolicy: IfNotPresent
            name: create
            securityContext:
              allowPrivilegeEscalation: false
              capabilities:
                drop:
                - ALL
              readOnlyRootFilesystem: true
              runAsGroup: 65532
              runAsNonRoot: true
              runAsUser: 65532
              seccompProfile:
                type: RuntimeDefault
          nodeSelector:
            kubernetes.io/os: linux
          restartPolicy: OnFailure
          serviceAccountName: ingress-nginx-admission
      ttlSecondsAfterFinished: 0
    ---
    apiVersion: batch/v1
    kind: Job
    metadata:
      labels:
        app.kubernetes.io/component: admission-webhook
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.13.3
      name: ingress-nginx-admission-patch
      namespace: ingress-nginx
    spec:
      template:
        metadata:
          labels:
            app.kubernetes.io/component: admission-webhook
            app.kubernetes.io/instance: ingress-nginx
            app.kubernetes.io/name: ingress-nginx
            app.kubernetes.io/part-of: ingress-nginx
            app.kubernetes.io/version: 1.13.3
          name: ingress-nginx-admission-patch
        spec:
          automountServiceAccountToken: true
          containers:
          - args:
            - patch
            - --webhook-name=ingress-nginx-admission
            - --namespace=$(POD_NAMESPACE)
            - --patch-mutating=false
            - --secret-name=ingress-nginx-admission
            - --patch-failure-policy=Fail
            env:
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            image: registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.6.3@sha256:3d671cf20a35cd94efc5dcd484970779eb21e7938c98fbc3673693b8a117cf39
            imagePullPolicy: IfNotPresent
            name: patch
            securityContext:
              allowPrivilegeEscalation: false
              capabilities:
                drop:
                - ALL
              readOnlyRootFilesystem: true
              runAsGroup: 65532
              runAsNonRoot: true
              runAsUser: 65532
              seccompProfile:
                type: RuntimeDefault
          nodeSelector:
            kubernetes.io/os: linux
          restartPolicy: OnFailure
          serviceAccountName: ingress-nginx-admission
      ttlSecondsAfterFinished: 0
    ---
    apiVersion: networking.k8s.io/v1
    kind: IngressClass
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.13.3
      name: nginx
    spec:
      controller: k8s.io/ingress-nginx
    ---
    apiVersion: admissionregistration.k8s.io/v1
    kind: ValidatingWebhookConfiguration
    metadata:
      labels:
        app.kubernetes.io/component: admission-webhook
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.13.3
      name: ingress-nginx-admission
    webhooks:
    - admissionReviewVersions:
      - v1
      clientConfig:
        service:
          name: ingress-nginx-controller-admission
          namespace: ingress-nginx
          path: /networking/v1/ingresses
          port: 443
      failurePolicy: Fail
      matchPolicy: Equivalent
      name: validate.nginx.ingress.kubernetes.io
      rules:
      - apiGroups:
        - networking.k8s.io
        apiVersions:
        - v1
        operations:
        - CREATE
        - UPDATE
        resources:
        - ingresses
      sideEffects: None
    ```
    :::

4. 檢視 ingress nginx 執行所需之 images

    ```
    find ${HOME}/k8s/addon/ingress/ingress-nginx \
      -name "daemonSet.yaml" \
      -type f \
      -exec grep image: {} + | \
        tr -s " " | \
        cut -d " " -f 3 | \
        sort -u | \
        uniq | \
        tee -a ${HOME}/k8s/img/img-list.txt
    ```

    執行結果：

    ```
    registry.k8s.io/ingress-nginx/controller:v1.13.3@sha256:1b044f6dcac3afbb59e05d98463f1dec6f3d3fb99940bc12ca5d80270358e3bd
    registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.6.3@sha256:3d671cf20a35cd94efc5dcd484970779eb21e7938c98fbc3673693b8a117cf39
    ```

5. 將 ingress nginx images 備份到指定目錄

    ```
    for i in $(grep "ingress-nginx" ${HOME}/k8s/img/img-list.txt); do podman pull ${i}; podman tag ${i} ${i%@*}; done
    podman save -m $(grep "ingress-nginx" ${HOME}/k8s/img/img-list.txt | cut -d "@" -f 1) > ${HOME}/k8s/img/ingress-nginx-controller-${INGRESS_NGINX}.tar
    ```

### 1.9. 準備 logging operator `6.1.0` 所需之 images

1. 定義 logging operator 版本變數

    ```
    LOGGING_OPERATOR="6.1.0"
    ```

2. 下載 logging operator chart file

    ```
    helm pull oci://ghcr.io/kube-logging/helm-charts/logging-operator \
      --version ${LOGGING_OPERATOR} \
      -d ${HOME}/k8s/addon/logging/logging-operator/
    ```

3. 檢視 logging operator 執行所需之 images

    ```
    echo 'ghcr.io/kube-logging/logging-operator:6.1.0
    ghcr.io/axoflow/axosyslog:4.18.1
    ghcr.io/kube-logging/logging-operator/fluentd:6.1.0-full
    ghcr.io/kube-logging/logging-operator/syslog-ng-reloader:6.1.0
    ghcr.io/kube-logging/logging-operator/config-reloader:6.1.0
    ghcr.io/kube-logging/logging-operator/fluentd-drain-watch:6.1.0
    ghcr.io/kube-logging/logging-operator/node-exporter:6.1.0
    ghcr.io/axoflow/axosyslog-metrics-exporter:0.0.13' >> ${HOME}/k8s/img/img-list.txt
    ```

4. 將 logging operator images 備份到指定目錄

    ```
    for i in $(grep -E "kube-logging|axoflow" ${HOME}/k8s/img/img-list.txt); do podman pull ${i}; done
    podman save -m $(grep -E "kube-logging|axoflow" ${HOME}/k8s/img/img-list.txt) > ${HOME}/k8s/img/kube-logging-${LOGGING_OPERATOR}.tar
    ```

### 1.10. 將整個工作目錄打包並匯入全離線的主機

1. 壓縮出全離線安裝包

    ```
    cd ~; tar -zcvf k8s-v1.34.1.tar.gz k8s/
    ```

2. 檢視全離線安裝包

    ```
    ls -lh ${HOME}/k8s-v1.34.1.tar.gz
    ```

    執行結果：

    ```
    -rw-r--r--. 1 bigred bigred 1.5G Oct 17 17:25 /home/bigred/k8s-v1.34.1.tar.gz
    ```

3. 將檔案上傳至所有全離線的 K8s 叢集主機，包含可上傳 image 至 Harbor 的主機

    ```
    scp ${HOME}/k8s-v1.34.1.tar.gz user_name@vm_ipaddress:.
    ```

### 1.11. 將離線安裝所需之 Container Images 匯入 Harbor

:::info

先決條件：已安裝好 Harbor

:::

1. 連線到可將 image tar 檔上傳至 Harbor 的主機

    ```
    ssh user@ipaddress
    ```

2. 定義 Harbor FQDN

    ```
    HARBOR_FQDN=harbor.example.com
    ```

2. 下載 Harbor CA 憑證

    ```
    sudo curl -k \
      -o /etc/pki/ca-trust/source/anchors/harbor-ca.crt \
      https://${HARBOR_FQDN}/api/v2.0/systeminfo/getcert
    ```

3. 更新系統的 CA 信任清單，讓 HTTPS 連線時能信任此憑證

    ```
    sudo update-ca-trust
    ```

4. 設定 podman 信任 Harbor 自簽 CA 憑證

    ```
    sudo mkdir /etc/containers/certs.d/${HARBOR_FQDN}

    sudo curl -k \
      -o /etc/containers/certs.d/${HARBOR_FQDN}/ca.crt \
      https://${HARBOR_FQDN}/api/v2.0/systeminfo/getcert
    ```

5. 登入 harbor

    ```
    podman login ${HARBOR_FQDN}
    ```

    執行結果：

    ```
    Username: admin
    Password:
    Login Succeeded!
    ```
    
6. 上傳所有 container images 至 Harbor

    ```
    cd ~/k8s/img; ./imgctl.sh
    ```

    正確執行結果：

    ```
    Load All Images Successfully
    Currnt: 1/29 harbor.example.com/library/kube-apiserver:v1.34.1 push ok
    Currnt: 2/29 harbor.example.com/library/kube-controller-manager:v1.34.1 push ok
    Currnt: 3/29 harbor.example.com/library/kube-scheduler:v1.34.1 push ok
    Currnt: 4/29 harbor.example.com/library/kube-proxy:v1.34.1 push ok
    Currnt: 5/29 harbor.example.com/library/coredns/coredns:v1.12.1 push ok
    Currnt: 6/29 harbor.example.com/library/pause:3.10.1 push ok
    Currnt: 7/29 harbor.example.com/library/etcd:3.6.4-0 push ok
    Currnt: 8/29 harbor.example.com/library/ingress-nginx/controller:v1.13.3@sha256:1b044f6dcac3afbb59e05d98463f1dec6f3d3fb99940bc12ca5d80270358e3bd push ok
    Currnt: 9/29 harbor.example.com/library/ingress-nginx/kube-webhook-certgen:v1.6.3@sha256:3d671cf20a35cd94efc5dcd484970779eb21e7938c98fbc3673693b8a117cf39 push ok
    Currnt: 10/29 harbor.example.com/library/kube-vip/kube-vip:v1.0.1 push ok
    Currnt: 11/29 harbor.example.com/library/kube-vip/kube-vip-cloud-provider:v0.0.12 push ok
    Currnt: 12/29 harbor.example.com/library/metrics-server/metrics-server:v0.8.0 push ok
    Currnt: 13/29 harbor.example.com/library/fluent/fluentd-kubernetes-daemonset:v1.19.0-debian-syslog-amd64-1.1 push ok
    Currnt: 14/29 harbor.example.com/library/cilium/cilium-envoy:v1.34.7-1757592137-1a52bb680a956879722f48c591a2ca90f7791324@sha256:7932d656b63f6f866b6732099d33355184322123cfe1182e6f05175a3bc2e0e0 push ok
    Currnt: 15/29 harbor.example.com/library/cilium/cilium:v1.18.2@sha256:858f807ea4e20e85e3ea3240a762e1f4b29f1cb5bbd0463b8aa77e7b097c0667 push ok
    Currnt: 16/29 harbor.example.com/library/cilium/hubble-relay:v1.18.2@sha256:6079308ee15e44dff476fb522612732f7c5c4407a1017bc3470916242b0405ac push ok
    Currnt: 17/29 harbor.example.com/library/cilium/hubble-ui-backend:v0.13.3@sha256:db1454e45dc39ca41fbf7cad31eec95d99e5b9949c39daaad0fa81ef29d56953 push ok
    Currnt: 18/29 harbor.example.com/library/cilium/hubble-ui:v0.13.3@sha256:661d5de7050182d495c6497ff0b007a7a1e379648e60830dd68c4d78ae21761d push ok
    Currnt: 19/29 harbor.example.com/library/cilium/operator-generic:v1.18.2@sha256:cb4e4ffc5789fd5ff6a534e3b1460623df61cba00f5ea1c7b40153b5efb81805 push ok
    Currnt: 20/29 harbor.example.com/library/sig-storage/nfs-subdir-external-provisioner:v4.0.2 push ok
    Currnt: 21/29 harbor.example.com/library/library/busybox:stable push ok
    Currnt: 22/29 harbor.example.com/library/kube-logging/logging-operator:6.1.0 push ok
    Currnt: 23/29 harbor.example.com/library/axoflow/axosyslog:4.18.1 push ok
    Currnt: 24/29 harbor.example.com/library/kube-logging/logging-operator/fluentd:6.1.0-full push ok
    Currnt: 25/29 harbor.example.com/library/kube-logging/logging-operator/syslog-ng-reloader:6.1.0 push ok
    Currnt: 26/29 harbor.example.com/library/kube-logging/logging-operator/config-reloader:6.1.0 push ok
    Currnt: 27/29 harbor.example.com/library/kube-logging/logging-operator/fluentd-drain-watch:6.1.0 push ok
    Currnt: 28/29 harbor.example.com/library/kube-logging/logging-operator/node-exporter:6.1.0 push ok
    Currnt: 29/29 harbor.example.com/library/axoflow/axosyslog-metrics-exporter:0.0.13 push ok
    ```

---

## 2. 建置 HA K8s 前置作業

### 2.1. 先決條件

1. 已將 K8s 全離線安裝包(檔名:`k8s-v1.34.1.tar.gz`)匯入 K8s 叢集中的每一台主機
3. 已將 Harbor 自簽 root CA 憑證匯入匯入 K8s 叢集中的每一台 (Harbor 如果不是自簽 root CA 則不用做到此步驟)

### 2.2. 連線至 K8s 的第一台 Control-plane 節點

1. ssh 連線至第一台 Control-plane 節點

    ```
    ssh user@ipaddress
    ```

### 2.3. 關閉  Firewalld Systemd Service

1. 關閉 Firewalld Systemd Service

    ```
    sudo systemctl disable --now firewalld.service
    ```

### 2.4. 調整 SELinux 為 `permissive` mode

1. 調整 SELinux 為 `permissive` mode

    ```
    # Set SELinux in permissive mode (effectively disabling it)
    sudo setenforce 0
    sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
    ```

### 2.5. 確認 SWAP 已關閉

1. 確認 SWAP 已關閉

    ```
    free -mh
    ```

    執行結果：

    ```
                   total        used        free      shared  buff/cache   available
    Mem:           7.5Gi       530Mi       6.2Gi       8.0Mi       1.0Gi       7.0Gi
    Swap:             0B          0B          0B
    ```

    > 確認 swap 欄位的值都是 `0` B

### 2.6. 確認 Kernel 版本 >= 5.10

1. 確認 Kernel 版本 >= 5.10

    ```
    uname -r
    ```

    執行結果：

    ```
    5.14.0-570.52.1.el9_6.x86_64
    ```

### 2.7. 添加必要 Kernel Modules

1. 永久添加 `br_netfilter` 和 `overlay` Kernel Modules

    ```
    cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
    overlay
    br_netfilter
    EOF
    ```

2. 立即啟用 `br_netfilter` 和 `overlay` Kernel Modules

    ```
    sudo modprobe overlay
    sudo modprobe br_netfilter
    ```

3. 驗證已載入 `br_netfilter` 和 `overlay` Kernel Modules

    ```
    sudo lsmod | grep -E "overlay|br_netfilter"
    ```

    執行結果：

    ```
    br_netfilter           36864  0
    bridge                417792  1 br_netfilter
    overlay               229376  0
    ```

### 2.8. 添加必要 Kernel Parameters

1. 永久添加必要 Kernel Parameters

    ```
    cat <<EOF | sudo tee /etc/sysctl.d/99-k8s-cri.conf
    net.bridge.bridge-nf-call-iptables = 1 
    net.bridge.bridge-nf-call-ip6tables = 1 
    net.ipv4.ip_forward = 1 
    EOF
    ```

2. 立即載入 Kernel Parameters

    ```
    sudo sysctl --system
    ```

3. 驗證 `net.bridge.bridge-nf-call-iptables`, `net.bridge.bridge-nf-call-ip6tables`, 和 `net.ipv4.ip_forward` 的值都設成 `1`

    ```
    sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-ip6tables net.ipv4.ip_forward
    ```

    執行結果：

    ```
    net.bridge.bridge-nf-call-iptables = 1
    net.bridge.bridge-nf-call-ip6tables = 1
    net.ipv4.ip_forward = 1
    ```

### 2.9. 解壓縮 K8s 全離線安裝包

1. 設定 K8s 全離線安裝包所在的目錄變數

    ```
    PREFIX="$HOME"
    ```

2. 解壓縮 K8s 全離線安裝包
    ```
    tar -zxvf "$PREFIX"/k8s-v1.34.1.tar.gz
    ```

    執行結果：

    ```
    ...
    tar: k8s/img/ingress-nginx-controller-v1.13.3.tar: time stamp 2025-10-20 14:18:13 is 166476.992186816 s in the future
    k8s/config/
    tar: k8s/img: time stamp 2025-10-20 14:34:01 is 167424.992012128 s in the future
    ```

### 2.10. 安裝所有必要軟體套件

1. 安裝所有必要軟體套件

    ```
    sudo yum localinstall "$HOME"/k8s/pkg/kubernetes/*.rpm
    ```

### 2.11. 確認節點可以透過 DNS 解析到 harbor

1. 定義 Harbor FQDN 變數

    ```
    HARBOR_FQDN=harbor.example.com
    DOMAIN_NAME_SYSTEM_SERVER=172.20.6.11
    ```

2. 確認節點可以透過 DNS 解析到 harbor

    ```
    dig @$DOMAIN_NAME_SYSTEM_SERVER $HARBOR_FQDN +short
    ```

    執行結果：

    ```
    172.20.7.32
    ```

### 2.12. 啟動 CRI-O Service 並設為開機自動啟動

1. 啟動 CRI-O Service 並設為開機自動啟動

    ```
    sudo systemctl enable --now crio.service
    ```

2. 確認 CRI-O Service 狀態

    ```
    sudo systemctl status crio.service
    ```

    執行結果：

    ```
    ● crio.service - Container Runtime Interface for OCI (CRI-O)
         Loaded: loaded (/usr/lib/systemd/system/crio.service; enabled; preset: disabled)
         Active: active (running) since Sat 2025-10-18 16:48:59 CST; 1min 18s ago
           Docs: https://github.com/cri-o/cri-o
       Main PID: 5734 (crio)
          Tasks: 7
         Memory: 20.0M
            CPU: 126ms
         CGroup: /system.slice/crio.service
                 └─5734 /usr/bin/crio

    Oct 18 16:48:59 topgun-c1.kubeantony.com crio[5734]: time="2025-10-18T16:48:59.314737723+08:00" le…cher"
    Oct 18 16:48:59 topgun-c1.kubeantony.com crio[5734]: time="2025-10-18T16:48:59.314774402+08:00" le…face"
    Oct 18 16:48:59 topgun-c1.kubeantony.com crio[5734]: time="2025-10-18T16:48:59.314808872+08:00" le…bled"
    Oct 18 16:48:59 topgun-c1.kubeantony.com crio[5734]: time="2025-10-18T16:48:59.314814672+08:00" le…ated"
    Oct 18 16:48:59 topgun-c1.kubeantony.com crio[5734]: time="2025-10-18T16:48:59.314820552+08:00" le… NRI"
    Oct 18 16:48:59 topgun-c1.kubeantony.com crio[5734]: time="2025-10-18T16:48:59.314823302+08:00" le…p..."
    Oct 18 16:48:59 topgun-c1.kubeantony.com crio[5734]: time="2025-10-18T16:48:59.314825973+08:00" le…s..."
    Oct 18 16:48:59 topgun-c1.kubeantony.com crio[5734]: time="2025-10-18T16:48:59.314833213+08:00" le…tate"
    Oct 18 16:48:59 topgun-c1.kubeantony.com crio[5734]: time="2025-10-18T16:48:59.314882773+08:00" le…bled"
    Oct 18 16:48:59 topgun-c1.kubeantony.com systemd[1]: Started Container Runtime Interface for OCI (…I-O).
    Hint: Some lines were ellipsized, use -l to show in full.
    ```

### 2.13. 設定 private registry 驗證

1. 將 Harbor 自簽 ca 憑證上傳至主機家目錄
2. 設定 Harbor 自簽 ca 憑證所在的目錄和 CA 憑證檔名為變數

    ```
    CA_PREFIX="$HOME"
    CA_FILE_NAME="ca.pem"
    ```

3. 讓 RHEL 信任 Harbor 自簽 ca 憑證

    ```
    sudo cp "$CA_PREFIX"/"$CA_FILE_NAME" /etc/pki/ca-trust/source/anchors/harbor-ca.crt
    sudo update-ca-trust
    ```
3. 編輯 CRI-O 設定檔，告訴它去哪裡讀取全域的認證資訊

    ```
    sudo nano /etc/crio/crio.conf.d/10-crio.conf
    ```

    檔案內容：
    > 新增 `global_auth_file` 那一行：指定 CRI-O 讀取 Docker 格式的全域認證檔案路徑
    > 還有 `pause_image` 這行，如果沒指定這行預設還是會到 K8s 的 image repo 去拉
    > 還有 `conmon_cgroup` 和 `cgroup_manager` 也要指定在 `crio.runtime` 區塊底下

    ```
    [crio.image]
    signature_policy = "/etc/crio/policy.json"
    global_auth_file = "/etc/crio/registries.d/auth.json"
    pause_image="harbor.example.com/library/pause:3.10.1"

    [crio.runtime]
    default_runtime = "crun"
    conmon_cgroup = "pod"
    cgroup_manager = "systemd"
    ```

4. 建立存放認證檔案的目錄

    ```
    sudo mkdir -p /etc/crio/registries.d/
    ```

5. 設定 Harbor Registry 的 FQDN (網域名稱) 變數，方便後續使用

    ```
    HARBOR_FQDN=harbor.example.com
    ```

6. 使用 podman 登入 Harbor，並將產生的認證 token 儲存到 CRI-O 指定的 auth.json 檔案中

    ```
    sudo podman login $HARBOR_FQDN --authfile /etc/crio/registries.d/auth.json
    ```

7. 驗證認證檔案是否已成功建立

    ```
    sudo cat /etc/crio/registries.d/auth.json
    ```

    執行結果如下：

    ```
    {
            "auths": {
                    "harbor.example.com": {
                            "auth": "YWRtaW46SGFyYm9yMTIzNDU="
                    }
            }
    }
    ```
6. 編輯 system-wide 的 registry 設定檔，供 Podman, CRI-O, Skopeo 等工具共用

    ```
    sudo nano /etc/containers/registries.conf
    ```

    檔案內容如下：
    > 注意，Harbor FQDN 每個環境會不同。

    ```
    [[registry]]
    location = "harbor.example.com" 
    insecure = false 
    ```

7. 重新啟動 CRI-O 服務，使其載入所有新的設定 (10-crio.conf 和 registries.conf)

    ```
    sudo systemctl restart crio
    ```

8. 使用 crictl (CRI 的客戶端工具) 測試是否能成功從私有 Harbor 拉取映像檔

    ```
    sudo crictl pull ${HARBOR_FQDN}/library/library/busybox:stable
    ```

    成功執行結果：

    ```
    Image is up to date for harbor.example.com/library/library/busybox@sha256:f0d4f96113b0ebc3bc741b49cc158254ddd91bbba8dba20281b2dacd2da4a45f
    ```

9. 清理測試，移除剛剛拉取的映像檔

    ```
    sudo crictl rmi ${HARBOR_FQDN}/library/library/busybox:stable
    ```

### 2.14. 設定 kubelet service node ip

1. 指定節點 internal ip

    ```
    INTERFACE=ens18
    IPV4_IP=$(ip -4 a s $INTERFACE | awk '/inet / {print $2}' | cut -d'/' -f1)
    sudo sed -i \
      -e '/^KUBELET_EXTRA_ARGS=$/s/$/\"--node-ip='"$IPV4_IP"'\"/' \
      /etc/sysconfig/kubelet
    ```

2. 檢查是否正確修改

    ```
    cat /etc/sysconfig/kubelet
    ```

    執行結果:

    ```
    KUBELET_EXTRA_ARGS="--node-ip=172.20.6.111"
    ```

### 2.15. 在執行 kubeadm 之前啟用 kubelet 服務

1. 啟動 kubelet 服務

    ```
    sudo systemctl enable --now kubelet.service
    ```

### 2.16. 設定 NTP 校時

1. 編輯 chrony 設定檔

    ```
    sudo nano /etc/chrony.conf
    ```

    檔案內容如下：

    ```
    #pool 2.rhel.pool.ntp.org iburst
    server 172.20.6.249 iburst
    ```

2. 重啟 chronyd 服務

    ```
    sudo systemctl restart chronyd
    ```

3. 確認是否成功校時

    ```
    sudo chronyc sources
    ```

    正確執行結果：

    ```
    MS Name/IP address         Stratum Poll Reach LastRx Last sample
    ===============================================================================
    ^* 172.20.6.249                  3   6   377    20    -64us[ -237us] +/-   19ms
    ```

### 2.16. 在其他所有 K8s 節點重複執行所有步驟 2.2. ~ 2.16. 

---

## 3. 透過 kubeadm 開始建立 HA K8s

### 3.1. 設定 kube-vip (單台 Control-plane 可跳過此步驟)

1. 以可以執行 sudo 的 user 透過 ssh 連線至第一台 Control-plane node

    ```
    ssh sudo_user@ipaddderss
    ```
2. 將 kube-vip static pod 拷貝到 `/etc/kubernetes/manifests/`

    ```
    sudo cp "$HOME"/k8s/addon/kube-vip/ha-control-plane/kube-vip-static-pod.yaml /etc/kubernetes/manifests/kube-vip.yaml
    ```

2. 修改 kube-vip 的 Service Account，只需在第一台 Control-plane 需要修改

    ```
    sudo sed -i 's|path: /etc/kubernetes/admin.conf|path: /etc/kubernetes/super-admin.conf|g' \
      /etc/kubernetes/manifests/kube-vip.yaml
    ```

3. 設定 harbor fqdn

    ```
    HARBOR_FQDN=harbor.example.com
    ```

4. 修改 kube-vip 的 image repo

    ```
    sudo sed -i "s|image: ghcr.io|image: ${HARBOR_FQDN}/library|g" \
      /etc/kubernetes/manifests/kube-vip.yaml
    ```

```
$ sudo nano /etc/kubernetes/manifests/kube-vip.yaml
......
spec:
  containers:
......
    - name: vip_interface
      value: ens18
    - name: vip_subnet
      value: "24"
......
    - name: address
      value: 10.10.7.31
```
### 3.2. 編輯 kubeadm 初始化設定檔

1. 設定 kubeadm 初始化設定檔

    ```
    nano "$HOME"/k8s/config/kubeadm-init.yaml
    ```

    > 注意每個環境需設定以下項目:
    > * `localAPIEndpoint.advertiseAddress`: API Server 宣告(Advertise)給叢集其他成員的 IP 位址。
    > * `nodeRegistration.criSocket`: Kubelet 應使用的 CRI socket 路徑。
    > * `nodeRegistration.name`: 此節點在 Kubernetes 叢集中的唯一識別名稱 (通常使用主機名稱)。
    > * `clusterName`: K8s 叢集的名稱。
    > * `controllerManager.extraVolumes`: 掛載額外的磁碟區 (Volume) 到 `kube-controller-manager` Pod 中。
    > * `imageRepository`: 用於拉取 Kubernetes 核心組件 (如 API Server, etcd) images 的 Registry 位址。
    > * `kubernetesVersion`: 要部署的 Kubernetes 確切版本。
    > * `networking.serviceSubnet`: 分配給 Kubernetes Services 使用的虛擬 IP 位址範圍 (CIDR)。
    > * `networking.podSubnet`: 分配給叢集中所有 Pods 使用的 IP 位址範圍 (CIDR)。
    > * `proxy.disabled`: 停用節點上的 `kube-proxy` 組件 。
    > * `controlPlaneEndpoint`: HA 架構的 K8s LoadBalance IP，供所有節點和使用者存取 Kubernetes API Server 的 IP 或 FQDN。
    > * `imageMaximumGCAge`: 節點上未使用 images 在被垃圾回收 (GC) 前可保留的 **最長** 存留時間。
    > * `imageMinimumGCAge`: 節點上未使用 images 在被垃圾回收 (GC) 前必須保留的 **最短** 存留時間。
    > * `shutdownGracePeriod`: Kubelet 執行正常關機程序 (Node Shutdown) 的總寬限時間。
    > * `shutdownGracePeriodCriticalPods`: 在正常關機期間，Kubelet 等待「關鍵 Pods (Critical Pods)」優雅終止的專用寬限時間。
    > * `systemReserved.memory`: 為 OS 系統守護進程 (如 systemd, sshd) 保留的記憶體量，Kubelet 不會使用這部分。
    > * `kubeReserved.memory`: 為 Kubernetes 系統組件 (如 kubelet, 容器執行階段) 保留的記憶體量，Pod 無法使用這部分。
    
    檔案內容如下 :

    ```
    apiVersion: kubeadm.k8s.io/v1beta4
    kind: InitConfiguration
    localAPIEndpoint:
      advertiseAddress: 172.20.6.111
      bindPort: 6443
    nodeRegistration:
      criSocket: unix:///var/run/crio/crio.sock
      imagePullPolicy: IfNotPresent
      imagePullSerial: true
      name: topgun-c1.kubeantony.com
      taints: null
    timeouts:
      controlPlaneComponentHealthCheck: 4m0s
      discovery: 5m0s
      etcdAPICall: 2m0s
      kubeletHealthCheck: 4m0s
      kubernetesAPICall: 1m0s
      tlsBootstrap: 5m0s
      upgradeManifests: 5m0s
    ---
    apiServer:
      certSANs:
       - 127.0.0.1
    apiVersion: kubeadm.k8s.io/v1beta4
    caCertificateValidityPeriod: 87600h0m0s
    certificateValidityPeriod: 8760h0m0s
    certificatesDir: /etc/kubernetes/pki
    clusterName: topgun
    controllerManager:
      extraArgs:
       - name: bind-address
         value: "0.0.0.0"
       - name: secure-port
         value: "10257"
      extraVolumes:
        - name: tz-config
          hostPath: /etc/localtime
          mountPath: /etc/localtime
          readOnly: true
    dns:
      imageRepository: harbor.example.com/library/coredns
    encryptionAlgorithm: RSA-2048
    etcd:
      local:
        dataDir: /var/lib/etcd
        extraArgs:
          - name: listen-metrics-urls
            value: http://0.0.0.0:2381
    imageRepository: harbor.example.com/library
    kind: ClusterConfiguration
    kubernetesVersion: 1.34.1
    networking:
      dnsDomain: cluster.local
      serviceSubnet: 10.96.0.0/16
      podSubnet: 10.244.0.0/16
    proxy:
      disabled: true
    scheduler:
      extraArgs:
        - name: bind-address
          value: "0.0.0.0"
        - name: secure-port
          value: "10259"
    controlPlaneEndpoint: 172.20.6.110:6443
    ---
    apiVersion: kubelet.config.k8s.io/v1beta1
    authentication:
      anonymous:
        enabled: false
      webhook:
        cacheTTL: 0s
        enabled: true
      x509:
        clientCAFile: /etc/kubernetes/pki/ca.crt
    authorization:
      mode: Webhook
      webhook:
        cacheAuthorizedTTL: 0s
        cacheUnauthorizedTTL: 0s
    cgroupDriver: systemd
    clusterDNS:
    - 10.96.0.10
    clusterDomain: cluster.local
    containerRuntimeEndpoint: ""
    cpuManagerReconcilePeriod: 0s
    crashLoopBackOff: {}
    evictionPressureTransitionPeriod: 0s
    fileCheckFrequency: 0s
    healthzBindAddress: 127.0.0.1
    healthzPort: 10248
    httpCheckFrequency: 0s
    imageMaximumGCAge: "168h"
    imageMinimumGCAge: "2m0s"
    kind: KubeletConfiguration
    logging:
      flushFrequency: 0
      options:
        json:
          infoBufferSize: "0"
        text:
          infoBufferSize: "0"
      verbosity: 0
    memorySwap: {}
    nodeStatusReportFrequency: 0s
    nodeStatusUpdateFrequency: 0s
    rotateCertificates: true
    runtimeRequestTimeout: 0s
    shutdownGracePeriod: 30s
    shutdownGracePeriodCriticalPods: 10s
    staticPodPath: /etc/kubernetes/manifests
    streamingConnectionIdleTimeout: 0s
    syncFrequency: 0s
    volumeStatsAggPeriod: 0s
    maxPods: 110
    systemReserved:
      memory: "1Gi"
    kubeReserved:
      memory: "2Gi"
    ```

### 3.3. 初始化第一台 Kubernetes control plane

1. Dry run set up the Kubernetes control plane，用以檢視是否設定檔有誤

    ```
    sudo kubeadm init \
      --dry-run \
      --upload-certs \
      --config="$HOME"/k8s/config/kubeadm-init.yaml
    ```

    正確執行結果 :

    ```
    ...
    Then you can join any number of worker nodes by running the following on each as root:

    kubeadm join 172.20.6.110:6443 --token 0n2iwr.ugdsni61mrtuqv4l \
            --discovery-token-ca-cert-hash sha256:0e9dd65a4b3ba662a1a9b9f448cc1a632b808f863dd892d46fa4005b731da5bb
    ```

2. Set up the Kubernetes control plane

    ```
    sudo kubeadm init \
      --upload-certs \
      --config="$HOME"/k8s/config/kubeadm-init.yaml
    ```

    正確執行結果 :

    ```
    W1018 22:31:59.203382    8373 initconfiguration.go:332] error unmarshaling configuration schema.GroupVersionKind{Group:"kubeadm.k8s.io", Version:"v1beta4", Kind:"ClusterConfiguration"}: strict decoding error: unknown field "apiServer.timeoutForControlPlane"
    [init] Using Kubernetes version: v1.34.1
    [preflight] Running pre-flight checks
            [WARNING SystemVerification]: kernel release 5.14.0-570.52.1.el9_6.x86_64 is unsupported. Supported LTS versions from the 5.x series are 5.4, 5.10 and 5.15. Any 6.x version is also supported. For cgroups v2 support, the recommended version is 5.10 or newer
    [preflight] Pulling images required for setting up a Kubernetes cluster
    [preflight] This might take a minute or two, depending on the speed of your internet connection
    [preflight] You can also perform this action beforehand using 'kubeadm config images pull'
    [certs] Using certificateDir folder "/etc/kubernetes/pki"
    [certs] Generating "ca" certificate and key
    [certs] Generating "apiserver" certificate and key
    [certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local topgun-c1.kubeantony.com] and IPs [10.96.0.1 172.20.6.111 172.20.6.110 127.0.0.1]
    [certs] Generating "apiserver-kubelet-client" certificate and key
    [certs] Generating "front-proxy-ca" certificate and key
    [certs] Generating "front-proxy-client" certificate and key
    [certs] Generating "etcd/ca" certificate and key
    [certs] Generating "etcd/server" certificate and key
    [certs] etcd/server serving cert is signed for DNS names [localhost topgun-c1.kubeantony.com] and IPs [172.20.6.111 127.0.0.1 ::1]
    [certs] Generating "etcd/peer" certificate and key
    [certs] etcd/peer serving cert is signed for DNS names [localhost topgun-c1.kubeantony.com] and IPs [172.20.6.111 127.0.0.1 ::1]
    [certs] Generating "etcd/healthcheck-client" certificate and key
    [certs] Generating "apiserver-etcd-client" certificate and key
    [certs] Generating "sa" key and public key
    [kubeconfig] Using kubeconfig folder "/etc/kubernetes"
    [kubeconfig] Writing "admin.conf" kubeconfig file
    [kubeconfig] Writing "super-admin.conf" kubeconfig file
    [kubeconfig] Writing "kubelet.conf" kubeconfig file
    [kubeconfig] Writing "controller-manager.conf" kubeconfig file
    [kubeconfig] Writing "scheduler.conf" kubeconfig file
    [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
    [control-plane] Using manifest folder "/etc/kubernetes/manifests"
    [control-plane] Creating static Pod manifest for "kube-apiserver"
    [control-plane] Creating static Pod manifest for "kube-controller-manager"
    [control-plane] Creating static Pod manifest for "kube-scheduler"
    [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
    [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/instance-config.yaml"
    [patches] Applied patch of type "application/strategic-merge-patch+json" to target "kubeletconfiguration"
    [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
    [kubelet-start] Starting the kubelet
    [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests"
    [kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s
    [kubelet-check] The kubelet is healthy after 501.238442ms
    [control-plane-check] Waiting for healthy control plane components. This can take up to 4m0s
    [control-plane-check] Checking kube-apiserver at https://172.20.6.111:6443/livez
    [control-plane-check] Checking kube-controller-manager at https://0.0.0.0:10257/healthz
    [control-plane-check] Checking kube-scheduler at https://0.0.0.0:10259/livez
    [control-plane-check] kube-controller-manager is healthy after 1.860780055s
    [control-plane-check] kube-scheduler is healthy after 2.364803775s
    [control-plane-check] kube-apiserver is healthy after 4.00165287s
    [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
    [kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
    [upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
    [upload-certs] Using certificate key:
    681423a365cee2a5ff5b612831e42a97ff941c0c6ad0595298a6ff603b0fae5d
    [mark-control-plane] Marking the node topgun-c1.kubeantony.com as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
    [mark-control-plane] Marking the node topgun-c1.kubeantony.com as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
    [bootstrap-token] Using token: z6kmo1.qp9d9u4xcb85xmjl
    [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
    [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
    [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
    [bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
    [bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
    [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
    [kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
    [addons] Applied essential addon: CoreDNS

    Your Kubernetes control-plane has initialized successfully!

    To start using your cluster, you need to run the following as a regular user:

      mkdir -p $HOME/.kube
      sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
      sudo chown $(id -u):$(id -g) $HOME/.kube/config

    Alternatively, if you are the root user, you can run:

      export KUBECONFIG=/etc/kubernetes/admin.conf

    You should now deploy a pod network to the cluster.
    Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
      https://kubernetes.io/docs/concepts/cluster-administration/addons/

    You can now join any number of control-plane nodes running the following command on each as root:

      kubeadm join 172.20.6.110:6443 --token z6kmo1.qp9d9u4xcb85xmjl \
            --discovery-token-ca-cert-hash sha256:4ea08e749828d599c6e3aefb69be7bbfbfe47bdab702907464fe033f22547e50 \
            --control-plane --certificate-key 681423a365cee2a5ff5b612831e42a97ff941c0c6ad0595298a6ff603b0fae5d

    Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
    As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
    "kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

    Then you can join any number of worker nodes by running the following on each as root:

    kubeadm join 172.20.6.110:6443 --token z6kmo1.qp9d9u4xcb85xmjl \
            --discovery-token-ca-cert-hash sha256:4ea08e749828d599c6e3aefb69be7bbfbfe47bdab702907464fe033f22547e50
    ```

### 3.4. 設定 kubeConfig

1. 設定 kubeconfig

    ```
    mkdir -p $HOME/.kube
    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    sudo chown $(id -u):$(id -g) $HOME/.kube/config
    printf "\nsource <(kubectl completion bash)\nalias k=kubectl\ncomplete -o default -F __start_kubectl k\n" >> ~/.bashrc
    source ~/.bashrc
    ```

2. 第一台 control plan 安裝好後再將 `kube-vip` 權限調整回來
   ```
   $ sudo sed -i 's|path: /etc/kubernetes/super-admin.conf|path: /etc/kubernetes/admin.conf|g' /etc/kubernetes/manifests/kube-vip.yaml

   $ sudo systemctl daemon-reload

   $ sudo systemctl restart kubelet
   ```


### 3.5. 安裝 Cilium CNI


1. 安裝 helm 指令

    ```
    cd ${HOME}/k8s/addon/cni/cilium/
    tar -zxvf helm-v3.19.0-linux-amd64.tar.gz
    sudo mv linux-amd64/helm /usr/local/bin/
    ```
2. 安裝 cilium 與 hubble client 指令

    ```
    tar -zxvf cilium-linux-amd64.tar.gz -C /tmp
    tar -zxvf hubble-linux-amd64.tar.gz -C /tmp
    sudo mv /tmp/cilium /usr/bin/
    sudo mv /tmp/hubble /usr/bin/
    ```

3. 編輯 cilium 設定檔

    ```
    nano custom-values.yaml
    ```

    檔案內容：

    ```
    operator:
      image:
        repository: harbor.example.com/library/cilium/operator
        tag: v1.18.2
        useDigest: false
    image:
      repository: harbor.example.com/library/cilium/cilium
      tag: v1.18.2
      useDigest: false
    envoy:
      image:
        repository: harbor.example.com/library/cilium/cilium-envoy
        tag: v1.34.7-1757592137-1a52bb680a956879722f48c591a2ca90f7791324
        useDigest: false
    bpf:
      masquerade: true
    hubble:
      relay:
        enabled: true
        image:
          repository: harbor.example.com/library/cilium/hubble-relay
          tag: v1.18.2
          useDigest: false
      ui:
        enabled: true
        service:
          type: NodePort
        frontend:
          image:
            repository: harbor.example.com/library/cilium/hubble-ui
            tag: v0.13.3
            useDigest: false
        backend:
          image:
            repository: harbor.example.com/library/cilium/hubble-ui-backend
            tag: v0.13.3
            useDigest: false
    ipam:
      mode: kubernetes
    ipv4NativeRoutingCIDR: 10.244.0.0/16
    k8sServiceHost: 172.20.6.110
    k8sServicePort: 6443
    kubeProxyReplacement: true
    autoDirectNodeRoutes: true
    routingMode: native
    devices: ens18
    ```

4. 安裝 cilium

    ```
    helm -n kube-system install cilium ./cilium-1.18.2.tgz -f custom-values.yaml
    ```

6. 確認 cilium 已安裝成功

    ```
    cilium status
    ```

    執行結果：

    ```
        /¯¯\
     /¯¯\__/¯¯\    Cilium:             OK
     \__/¯¯\__/    Operator:           OK
     /¯¯\__/¯¯\    Envoy DaemonSet:    OK
     \__/¯¯\__/    Hubble Relay:       OK
        \__/       ClusterMesh:        disabled

    DaemonSet              cilium                   Desired: 6, Ready: 6/6, Available: 6/6
    DaemonSet              cilium-envoy             Desired: 6, Ready: 6/6, Available: 6/6
    Deployment             cilium-operator          Desired: 2, Ready: 2/2, Available: 2/2
    Deployment             hubble-relay             Desired: 1, Ready: 1/1, Available: 1/1
    Deployment             hubble-ui                Desired: 1, Ready: 1/1, Available: 1/1
    Containers:            cilium                   Running: 6
                           cilium-envoy             Running: 6
                           cilium-operator          Running: 2
                           clustermesh-apiserver
                           hubble-relay             Running: 1
                           hubble-ui                Running: 1
    Cluster Pods:          9/9 managed by Cilium
    Helm chart version:
    Image versions         cilium             harbor.example.com/library/cilium/cilium:v1.18.2: 6
                           cilium-envoy       harbor.example.com/library/cilium/cilium-envoy:v1.34.7-1757592137-1a52bb680a956879722f48c591a2ca90f7791324: 6
                           cilium-operator    harbor.example.com/library/cilium/operator-generic:v1.18.2: 2
                           hubble-relay       harbor.example.com/library/cilium/hubble-relay:v1.18.2: 1
                           hubble-ui          harbor.example.com/library/cilium/hubble-ui-backend:v0.13.3: 1
                           hubble-ui          harbor.example.com/library/cilium/hubble-ui:v0.13.3: 1
    ```

7. 確認 cilium 有啟用 eBPF host routing

    ```
    cilium_pod=$(kubectl -n kube-system get pods --field-selector status.phase=Running -l k8s-app=cilium -o name | tail -n 1 | cut -d "/" -f 2)
    kubectl -n kube-system exec -it ${cilium_pod} -- cilium-dbg status
    ```

    執行結果：

    ```
    KubeProxyReplacement:    True   [ens18    172.20.6.115 172.20.6.117 fe80::be24:11ff:fe2e:cc62 (Direct Routing)]
    Host firewall:           Disabled
    SRv6:                    Disabled
    CNI Chaining:            none
    CNI Config file:         successfully wrote CNI configuration file to /host/etc/cni/net.d/05-cilium.conflist
    Cilium:                  Ok   1.18.2 (v1.18.2-5bd307a8)
    NodeMonitor:             Listening for events on 2 CPUs with 64x4096 of shared memory
    Cilium health daemon:    Ok
    IPAM:                    IPv4: 7/254 allocated from 10.244.5.0/24,
    IPv4 BIG TCP:            Disabled
    IPv6 BIG TCP:            Disabled
    BandwidthManager:        Disabled
    Routing:                 Network: Native   Host: BPF
    Attach Mode:             TCX
    Device Mode:             veth
    Masquerading:            BPF   [ens18]   10.244.0.0/16  [IPv4: Enabled, IPv6: Disabled]
    Controller Status:       47/47 healthy
    Proxy Status:            OK, ip 10.244.5.177, 0 redirects active on ports 10000-20000, Envoy: external
    Global Identity Range:   min 256, max 65535
    Hubble:                  Ok              Current/Max Flows: 2491/4095 (60.83%), Flows/s: 3.51   Metrics: Disabled
    Encryption:              Disabled
    Cluster health:          6/6 reachable   (2025-10-23T08:51:12Z)   (Probe interval: 2m33.896961447s)
    Name                     IP              Node                     Endpoints
    Modules Health:          Stopped(12) Degraded(0) OK(78)
    ```

### 3.6. 新增其他 Control-plane 節點進 K8s 叢集 (單台 Control-plane 可跳過此步驟)

1. 以可以執行 sudo 的 user 透過 ssh 連線至其他 Control-plane node

    ```
    ssh sudo_user@ipaddderss
    ```

2. 將 kube-vip static pod 拷貝到 `/etc/kubernetes/manifests/`

    ```
    sudo cp "$HOME"/k8s/addon/kube-vip/ha-control-plane/kube-vip-static-pod.yaml /etc/kubernetes/manifests/kube-vip.yaml
    ```
3. 設定 harbor fqdn

    ```
    HARBOR_FQDN=harbor.example.com
    ```

4. 修改 kube-vip 的 image repo

    ```
    sudo sed -i "s|image: ghcr.io|image: ${HARBOR_FQDN}/library|g" \
      /etc/kubernetes/manifests/kube-vip.yaml
    ```
5. 獲得 Control-plane node Kubeadm Join Command
    請在第一台 Control-plane node 執行以下命令

    ```
    kubeadm_init_config_file="${HOME}/k8s/config/kubeadm-init.yaml"

    certificate_key=$(sudo kubeadm init phase upload-certs --upload-certs --config ${kubeadm_init_config_file} | tail -n 1)

    join_cmd=$(sudo kubeadm token create --print-join-command)

    join_cp_cmd="sudo $join_cmd --control-plane --certificate-key $certificate_key"

    echo $join_cp_cmd
    ```
    
    執行結果：

    ```
    sudo kubeadm join 172.20.6.110:6443 --token rl5cdl.8552g1j3ezrvxk21 --discovery-token-ca-cert-hash sha256:4ea08e749828d599c6e3aefb69be7bbfbfe47bdab702907464fe033f22547e50 --control-plane --certificate-key d52af404ed5765485f28e0c5e7e886ae244a8861a78dc3b1ce1a0487194e8a1a
    ```

6. 將 Control-plane 節點加入 K8s 叢集
    請在要加入 K8s 的 Control-plane node 執行以下命令

    ```
    sudo kubeadm join 172.20.6.110:6443 --token rl5cdl.8552g1j3ezrvxk21 --discovery-token-ca-cert-hash sha256:4ea08e749828d599c6e3aefb69be7bbfbfe47bdab702907464fe033f22547e50 --control-plane --certificate-key d52af404ed5765485f28e0c5e7e886ae244a8861a78dc3b1ce1a0487194e8a1a
    ```
    > join node 使用的 token 會在 24 小時後自動過期

7. 重複以上 1. ~ 6. 步驟將其他 Control-plane node 加到 K8s Cluster

 ### 3.7. 新增 Worker 節點進 K8s 叢集
 
 1. **在第一台 Controlplane** 將 join worker node 指令 print 出來

    ```
    sudo kubeadm token create --print-join-command
    ```

    執行結果：

    ```
    kubeadm join 172.20.6.110:6443 --token inyh77.qp4dqe6wto9uhk8m --discovery-token-ca-cert-hash sha256:4ea08e749828d599c6e3aefb69be7bbfbfe47bdab702907464fe033f22547e50
    ```

2. 以可以執行 sudo 的 user 透過 ssh 連線至 worker node

    ```
    ssh sudo_user@ipaddderss
    ```

3. 將 worker node 加入 K8s 叢集

    ```
    sudo kubeadm join 172.20.6.110:6443 \
      --token inyh77.qp4dqe6wto9uhk8m \
      --discovery-token-ca-cert-hash sha256:4ea08e749828d599c6e3aefb69be7bbfbfe47bdab702907464fe033f22547e50
    ```
4. ssh 連線到其他 worker node，重複執行步驟 3，將其他 worker node 也加入 K8s 叢集
5. 回到第一台 Control-plane 執行以下命令，將所有 worker nodes 都貼 label

    ```
    kubectl label node topgun-w1.kubeantony.com node-role.kubernetes.io/worker=
    kubectl label node topgun-w2.kubeantony.com node-role.kubernetes.io/worker=
    kubectl label node topgun-w3.kubeantony.com node-role.kubernetes.io/worker=
    ```

### 3.8. 檢視 K8s 狀態是否正常

1. 檢視 K8s node 狀態

    ```
    kubectl get nodes
    ```

    執行結果：

    ```
    NAME                       STATUS   ROLES           AGE     VERSION
    topgun-c1.kubeantony.com   Ready    control-plane   3d15h   v1.34.1
    topgun-c2.kubeantony.com   Ready    control-plane   26h     v1.34.1
    topgun-c3.kubeantony.com   Ready    control-plane   3h24m   v1.34.1
    topgun-w1.kubeantony.com   Ready    worker          3d6h    v1.34.1
    topgun-w2.kubeantony.com   Ready    worker          17m     v1.34.1
    topgun-w3.kubeantony.com   Ready    worker          16m     v1.34.1
    ```

2. 檢視 kube-system namespace 底下所有 pods 狀態

    ```
    kubectl -n kube-system get pods
    ```

    執行結果：

    ```
    NAME                                               READY   STATUS    RESTARTS      AGE
    cilium-b74sh                                       1/1     Running   0             26h
    cilium-btg2h                                       1/1     Running   0             26h
    cilium-envoy-24skx                                 1/1     Running   0             26h
    cilium-envoy-llwv7                                 1/1     Running   0             17m
    cilium-envoy-nb5l2                                 1/1     Running   0             3h24m
    cilium-envoy-nvkz9                                 1/1     Running   0             26h
    cilium-envoy-vz6kj                                 1/1     Running   0             16m
    cilium-envoy-wm4xw                                 1/1     Running   0             26h
    cilium-gp9v9                                       1/1     Running   0             17m
    cilium-operator-794b864946-8j8j5                   1/1     Running   1 (10h ago)   26h
    cilium-operator-794b864946-9whct                   1/1     Running   0             26h
    cilium-vqp5v                                       1/1     Running   0             26h
    cilium-xgcch                                       1/1     Running   0             16m
    cilium-zj2xj                                       1/1     Running   0             3h24m
    coredns-f74588c75-86h95                            1/1     Running   0             26h
    coredns-f74588c75-9992t                            1/1     Running   0             26h
    etcd-topgun-c1.kubeantony.com                      1/1     Running   0             3d15h
    etcd-topgun-c2.kubeantony.com                      1/1     Running   0             26h
    etcd-topgun-c3.kubeantony.com                      1/1     Running   0             3h24m
    hubble-relay-7d6865bf7b-7wvhl                      1/1     Running   0             26h
    hubble-ui-c6998f8ff-6m4cn                          2/2     Running   0             26h
    kube-apiserver-topgun-c1.kubeantony.com            1/1     Running   0             3d15h
    kube-apiserver-topgun-c2.kubeantony.com            1/1     Running   0             26h
    kube-apiserver-topgun-c3.kubeantony.com            1/1     Running   0             3h24m
    kube-controller-manager-topgun-c1.kubeantony.com   1/1     Running   1 (10h ago)   3d15h
    kube-controller-manager-topgun-c2.kubeantony.com   1/1     Running   0             26h
    kube-controller-manager-topgun-c3.kubeantony.com   1/1     Running   0             3h24m
    kube-scheduler-topgun-c1.kubeantony.com            1/1     Running   1 (10h ago)   3d15h
    kube-scheduler-topgun-c2.kubeantony.com            1/1     Running   0             26h
    kube-scheduler-topgun-c3.kubeantony.com            1/1     Running   0             3h24m
    kube-vip-topgun-c1.kubeantony.com                  1/1     Running   1 (10h ago)   3d15h
    kube-vip-topgun-c3.kubeantony.com                  1/1     Running   0             3h24m
    ```

### 3.9. 檢測 K8s 網路延遲和吞吐量

1. 主要測試以下項目
    - **吞吐量 - TCP Throughput (TCP_STREAM)**：有助於了解在特定配置下可達到的最大吞吐量。
        - 此測試代表大量資料傳輸的應用程式，例如串流服務或執行資料上傳/下載的服務。
    - **請求/回應速率 - Request/Response Rate (TCP_RR)**：主要測量處理單一網路封包來回轉發的延遲與效率。這項基準測試將在線路上產生每秒可能的最大封包數（packets per second），並著重測試處理單一網路封包的成本（資源消耗）。
        - 此測試代表了那些維持持續性連線（persistent connections）、並與其他服務交換請求/回應類型互動的服務。這對於使用 REST 或 gRPC API 的服務來說很常見。
    - **連線速率 - Connection Rate (TCP_CRR)**：它類似於 Request/Response Rate 測試，但會為每一次來回建立一個新的 TCP 連線。
        - 此測試代表暴露於公網的服務，它會接收來自許多客戶端的連線。L4 代理或對外部端點開啟許多連線的服務，都是很好的例子。

1. 透過 cilium 指令檢測 K8s 網路頻寬

    ```
    cilium connectivity perf \
      --crr \
      --host-to-pod \
      --pod-to-host \
      --performance-image quay.io/cilium/network-perf:1751527436-c2462ae \
      --node-selector-client kubernetes.io/hostname=topgun-w1.kubeantony.com \
      --node-selector-server kubernetes.io/hostname=topgun-w2.kubeantony.com
    ```

    執行結果如下：

    ```
    🏃[cilium-test-1] Running 1 tests ...
    [=] [cilium-test-1] Test [network-perf] [1/1]
    ................................

    🔥 Network Performance Test Summary [cilium-test-1]:
    --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    📋 Scenario        | Node       | Test            | Duration        | Min             | Mean            | Max             | P50             | P90             | P99             | Transaction rate OP/s
    --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    📋 pod-to-pod      | same-node  | TCP_CRR         | 10s             | 50µs            | 84.48µs         | 11.99ms         | 77µs            | 96µs            | 144µs           | 11787.95
    📋 pod-to-pod      | same-node  | TCP_RR          | 10s             | 11µs            | 30.74µs         | 4.659ms         | 31µs            | 35µs            | 46µs            | 32197.86
    📋 pod-to-host     | same-node  | TCP_CRR         | 10s             | 61µs            | 98.57µs         | 21.094ms        | 90µs            | 109µs           | 155µs           | 10104.54
    📋 pod-to-host     | same-node  | TCP_RR          | 10s             | 12µs            | 31.86µs         | 1.091ms         | 32µs            | 37µs            | 49µs            | 31071.90
    📋 host-to-pod     | same-node  | TCP_CRR         | 10s             | 60µs            | 100.33µs        | 14.059ms        | 90µs            | 109µs           | 160µs           | 9930.36
    📋 host-to-pod     | same-node  | TCP_RR          | 10s             | 12µs            | 31.94µs         | 892µs           | 32µs            | 36µs            | 48µs            | 30996.63
    📋 host-to-host    | same-node  | TCP_CRR         | 10s             | 52µs            | 76.41µs         | 1.924ms         | 73µs            | 89µs            | 127µs           | 13016.02
    📋 host-to-host    | same-node  | TCP_RR          | 10s             | 12µs            | 31.89µs         | 1.281ms         | 32µs            | 36µs            | 48µs            | 31043.05
    📋 pod-to-pod      | other-node | TCP_CRR         | 10s             | 242µs           | 330.04µs        | 13.257ms        | 301µs           | 342µs           | 790µs           | 3027.08
    📋 pod-to-pod      | other-node | TCP_RR          | 10s             | 87µs            | 112.64µs        | 4.085ms         | 112µs           | 124µs           | 146µs           | 8867.04
    📋 pod-to-host     | other-node | TCP_CRR         | 10s             | 243µs           | 321.36µs        | 15.197ms        | 302µs           | 340µs           | 483µs           | 3108.95
    📋 pod-to-host     | other-node | TCP_RR          | 10s             | 89µs            | 113.52µs        | 3.172ms         | 113µs           | 125µs           | 147µs           | 8797.96
    📋 host-to-pod     | other-node | TCP_CRR         | 10s             | 230µs           | 311.48µs        | 13.035ms        | 287µs           | 334µs           | 635µs           | 3207.10
    📋 host-to-pod     | other-node | TCP_RR          | 10s             | 88µs            | 115.57µs        | 4.126ms         | 114µs           | 128µs           | 153µs           | 8641.95
    📋 host-to-host    | other-node | TCP_CRR         | 10s             | 229µs           | 303.13µs        | 21.099ms        | 285µs           | 322µs           | 417µs           | 2739.19
    📋 host-to-host    | other-node | TCP_RR          | 10s             | 89µs            | 114.78µs        | 4.428ms         | 114µs           | 126µs           | 148µs           | 8702.07
    --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    ----------------------------------------------------------------------------------------
    📋 Scenario        | Node       | Test               | Duration        | Throughput Mb/s
    ----------------------------------------------------------------------------------------
    📋 pod-to-pod      | same-node  | TCP_STREAM         | 10s             | 18952.05
    📋 pod-to-pod      | same-node  | TCP_STREAM_MULTI   | 10s             | 45634.84
    📋 pod-to-host     | same-node  | TCP_STREAM         | 10s             | 16338.34
    📋 pod-to-host     | same-node  | TCP_STREAM_MULTI   | 10s             | 41785.81
    📋 host-to-pod     | same-node  | TCP_STREAM         | 10s             | 15643.64
    📋 host-to-pod     | same-node  | TCP_STREAM_MULTI   | 10s             | 40812.09
    📋 host-to-host    | same-node  | TCP_STREAM         | 10s             | 36117.05
    📋 host-to-host    | same-node  | TCP_STREAM_MULTI   | 10s             | 82274.99
    📋 pod-to-pod      | other-node | TCP_STREAM         | 10s             | 9407.21
    📋 pod-to-pod      | other-node | TCP_STREAM_MULTI   | 10s             | 9417.40
    📋 pod-to-host     | other-node | TCP_STREAM         | 10s             | 9411.27
    📋 pod-to-host     | other-node | TCP_STREAM_MULTI   | 10s             | 9411.89
    📋 host-to-pod     | other-node | TCP_STREAM         | 10s             | 9409.00
    📋 host-to-pod     | other-node | TCP_STREAM_MULTI   | 10s             | 9398.15
    📋 host-to-host    | other-node | TCP_STREAM         | 10s             | 9408.12
    📋 host-to-host    | other-node | TCP_STREAM_MULTI   | 10s             | 9416.04
    ----------------------------------------------------------------------------------------

    ✅ [cilium-test-1] All 1 tests (32 actions) successful, 0 tests skipped, 0 scenarios skipped.
    ```

    未開 ebpf host routing 的數值：

    ```
    ...
    🏃[cilium-test-1] Running 1 tests ...
    [=] [cilium-test-1] Test [network-perf] [1/1]
    ................................

    🔥 Network Performance Test Summary [cilium-test-1]:
    --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    📋 Scenario        | Node       | Test            | Duration        | Min             | Mean            | Max             | P50             | P90             | P99             | Transaction rate OP/s
    --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    📋 pod-to-pod      | same-node  | TCP_CRR         | 10s             | 50µs            | 82.85µs         | 10.656ms        | 78µs            | 94µs            | 124µs           | 12017.63
    📋 pod-to-pod      | same-node  | TCP_RR          | 10s             | 10µs            | 30.71µs         | 828µs           | 30µs            | 35µs            | 46µs            | 32231.92
    📋 pod-to-host     | same-node  | TCP_CRR         | 10s             | 59µs            | 94.56µs         | 12.326ms        | 91µs            | 108µs           | 137µs           | 10532.25
    📋 pod-to-host     | same-node  | TCP_RR          | 10s             | 12µs            | 31.94µs         | 2.572ms         | 32µs            | 37µs            | 48µs            | 30996.44
    📋 host-to-pod     | same-node  | TCP_CRR         | 10s             | 56µs            | 93.13µs         | 15.588ms        | 90µs            | 107µs           | 136µs           | 10694.60
    📋 host-to-pod     | same-node  | TCP_RR          | 10s             | 12µs            | 31.27µs         | 1.842ms         | 31µs            | 36µs            | 48µs            | 31651.76
    📋 host-to-host    | same-node  | TCP_CRR         | 10s             | 53µs            | 75µs            | 1.479ms         | 72µs            | 88µs            | 123µs           | 13261.02
    📋 host-to-host    | same-node  | TCP_RR          | 10s             | 12µs            | 31.87µs         | 583µs           | 32µs            | 36µs            | 47µs            | 31059.78
    📋 pod-to-pod      | other-node | TCP_CRR         | 10s             | 278µs           | 368.98µs        | 9.809ms         | 339µs           | 382µs           | 965µs           | 2707.98
    📋 pod-to-pod      | other-node | TCP_RR          | 10s             | 95µs            | 123.71µs        | 4.119ms         | 122µs           | 137µs           | 163µs           | 8073.77
    📋 pod-to-host     | other-node | TCP_CRR         | 10s             | 248µs           | 371.53µs        | 1.052362s       | 298µs           | 335µs           | 500µs           | 2006.19
    📋 pod-to-host     | other-node | TCP_RR          | 10s             | 90µs            | 114.49µs        | 2.982ms         | 114µs           | 126µs           | 147µs           | 8723.87
    📋 host-to-pod     | other-node | TCP_CRR         | 10s             | 279µs           | 366.76µs        | 12.3ms          | 338µs           | 375µs           | 850µs           | 2724.34
    📋 host-to-pod     | other-node | TCP_RR          | 10s             | 105µs           | 135.79µs        | 15.723ms        | 133µs           | 148µs           | 176µs           | 7355.48
    📋 host-to-host    | other-node | TCP_CRR         | 10s             | 229µs           | 301.42µs        | 15.967ms        | 284µs           | 314µs           | 415µs           | 2739.19
    📋 host-to-host    | other-node | TCP_RR          | 10s             | 88µs            | 114.62µs        | 4.249ms         | 114µs           | 127µs           | 148µs           | 8713.25
    --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    ----------------------------------------------------------------------------------------
    📋 Scenario        | Node       | Test               | Duration        | Throughput Mb/s
    ----------------------------------------------------------------------------------------
    📋 pod-to-pod      | same-node  | TCP_STREAM         | 10s             | 19819.33
    📋 pod-to-pod      | same-node  | TCP_STREAM_MULTI   | 10s             | 44767.93
    📋 pod-to-host     | same-node  | TCP_STREAM         | 10s             | 17519.15
    📋 pod-to-host     | same-node  | TCP_STREAM_MULTI   | 10s             | 41288.42
    📋 host-to-pod     | same-node  | TCP_STREAM         | 10s             | 15667.62
    📋 host-to-pod     | same-node  | TCP_STREAM_MULTI   | 10s             | 40686.99
    📋 host-to-host    | same-node  | TCP_STREAM         | 10s             | 36663.82
    📋 host-to-host    | same-node  | TCP_STREAM_MULTI   | 10s             | 83454.15
    📋 pod-to-pod      | other-node | TCP_STREAM         | 10s             | 5515.69
    📋 pod-to-pod      | other-node | TCP_STREAM_MULTI   | 10s             | 5462.09
    📋 pod-to-host     | other-node | TCP_STREAM         | 10s             | 9393.23
    📋 pod-to-host     | other-node | TCP_STREAM_MULTI   | 10s             | 9399.02
    📋 host-to-pod     | other-node | TCP_STREAM         | 10s             | 5456.02
    📋 host-to-pod     | other-node | TCP_STREAM_MULTI   | 10s             | 5423.52
    📋 host-to-host    | other-node | TCP_STREAM         | 10s             | 9410.65
    📋 host-to-host    | other-node | TCP_STREAM_MULTI   | 10s             | 9416.39
    ----------------------------------------------------------------------------------------

    ✅ [cilium-test-1] All 1 tests (32 actions) successful, 0 tests skipped, 0 scenarios skipped.
    ```

- 通用欄位
    * `Scenario` (情境):
        描述流量的來源和目的地。
        * `pod-to-pod`: 從一個 Pod 到另一個 Pod。
        * `pod-to-host`: 從一個 Pod 到其運行的節點 (Host)。
        * `host-to-pod`: 從一個節點 (Host) 到一個 Pod。
        * `host-to-host`: 從一個節點 (Host) 到另一個節點 (Host)。
    * **`Node` (節點):**
        描述來源和目的地的相對位置。
        * `same-node`: 來源和目的地在**同一個** Kubernetes 節點上。
        * `other-node`: 來源和目的地在**不同的** Kubernetes 節點上。
- 表 1
    這張表格專注於延遲（完成一個操作需要多長時間）和事務處理速率（每秒可以完成多少操作），主要用於 `TCP_CRR` 和 `TCP_RR` 測試。時間單位是微秒 (`µs`) 或毫秒 (`ms`)。
    * `Min` (最小延遲): 在測試期間觀察到的最快的單次事務（請求/回應或連接/關閉）延遲。
    * `Mean` (平均延遲): 所有事務延遲的算術平均值。
    * `Max` (最大延遲): 在測試期間觀察到的最慢的單次事務延遲。高 `Max` 值可能表示網路抖動或偶發性的效能瓶頸。
    * `P50` (50百分位延遲): 也稱為中位數。50% 的事務延遲低於此值。與 `Mean` 相比，`P50` 受極端異常值的影響較小。
    * `P90` (90百分位延遲): 90% 的事務延遲低於此值。這是一個常見的服務水平指標 (SLI)，用於衡量絕大多數用戶的體驗。
    * `P99` (99百分位延遲): 99% 的事務延遲低於此值。這有助於識別「長尾延遲」(long-tail latency)，即影響少數請求的嚴重效能問題。
    * `Transaction rate OP/s` (每秒事務處理速率): 每秒操作數 (Operations Per Second)。
        * 這個值越高越好，表示處理事務的效率越高。
- 表 2
這張表格專注於頻寬（每秒可以傳輸多少數據），主要用於 `TCP_STREAM` 和 `TCP_STREAM_MULTI` 測試。
    * `Throughput Mb/s` (吞吐量 Mb/s):
        - 每秒百萬位元 (Megabits per second)。這個值顯示了在測試期間，來源和目的地之間數據傳輸的平均速率。值越高，表示網路頻寬越大。
    * `TCP_STREAM`: 
        - 使用單個 TCP 連接測量最大數據傳輸吞吐量（頻寬）。
    * `TCP_STREAM_MULTI`: 
        - 使用 32 個並行 TCP 連接測量最大數據傳輸吞吐量。這通常能更好地佔滿網路鏈路，顯示系統的總頻寬上限。

---

## 4. 安裝 K8s Addon

### 4.1. 安裝 Metrics server

1. 切換工作目錄

    ```
    cd ${HOME}/k8s/addon/metrics/metrics-server
    ```

2. 編輯 Kustomization File

    ```
    nano kustomization.yaml
    ```

    檔案內容：

    ```
    apiVersion: kustomize.config.k8s.io/v1beta1
    kind: Kustomization
    resources:
    - ./components.yaml

    images:
    - name: registry.k8s.io/metrics-server/metrics-server
      newName: harbor.example.com/library/metrics-server/metrics-server
      newTag: v0.8.0

    patches:
    - target:
        kind: Deployment
        name: metrics-server
      patch: |-
        - op: add
          path: /spec/template/spec/containers/0/args/-
          value: --kubelet-insecure-tls
    ```

3. 驗證 yaml 設定檔是否有誤

    ```
    kubectl apply -k . --dry-run=server
    ```

    正確執行結果：

    ```
    serviceaccount/metrics-server created (server dry run)
    clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created (server dry run)
    clusterrole.rbac.authorization.k8s.io/system:metrics-server created (server dry run)
    rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created (server dry run)
    clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created (server dry run)
    clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created (server dry run)
    service/metrics-server created (server dry run)
    deployment.apps/metrics-server created (server dry run)
    apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created (server dry run)
    ```

4. 部署 Metrics Server

    ```
    kubectl apply -k .
    ```

5. 檢視 Metrics Server pods 運作狀態

    ```
    kubectl -n kube-system get pods -l k8s-app=metrics-server
    ```

    執行結果：

    ```
    NAME                             READY   STATUS    RESTARTS   AGE
    metrics-server-69779649f-frvqz   1/1     Running   0          43s
    ```

6. 檢視 K8s 所有 nodes 的 cpu 記憶體使用狀況

    ```
    kubectl top nodes
    ```

    執行結果：

    ```
    NAME                       CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)
    topgun-c1.kubeantony.com   41m          2%       1705Mi          37%
    topgun-c2.kubeantony.com   34m          1%       1175Mi          26%
    topgun-c3.kubeantony.com   28m          1%       1057Mi          23%
    topgun-w1.kubeantony.com   12m          0%       1066Mi          23%
    topgun-w2.kubeantony.com   10m          0%       947Mi           21%
    topgun-w3.kubeantony.com   9m           0%       778Mi           17%
    ```

### 4.2. 安裝 kube-vip On-Premises Cloud Controller

1. 切換工作目錄

    ```
    cd ${HOME}/k8s/addon/kube-vip/k8s-lb-service
    ```

2. 將可分配的 IP Range 寫入 configmap

    ```
    kubectl -n kube-system create configmap kubevip \
      --from-literal range-global=172.20.6.117-172.20.6.120 \
      --dry-run=client -o yaml > kube-vip-cm.yaml
    ```

3. 編輯 Kustomization File

    ```
    nano kustomization.yaml
    ```

    檔案內容：

    ```
    apiVersion: kustomize.config.k8s.io/v1beta1
    kind: Kustomization
    resources:
    - ./kube-vip-rbac.yaml
    - ./kube-vip-ds.yaml
    - ./kube-vip-cloud-controller.yaml
    - ./kube-vip-cm.yaml

    images:
    - name: ghcr.io/kube-vip/kube-vip
      newName: harbor.example.com/library/kube-vip/kube-vip
      newTag: v1.0.1
    - name: ghcr.io/kube-vip/kube-vip-cloud-provider
      newName: harbor.example.com/library/kube-vip/kube-vip-cloud-provider
      newTag: v0.0.12

    patches:
    - target:
        kind: DaemonSet
        name: kube-vip-ds
        namespace: kube-system
      patch: |-
        apiVersion: apps/v1
        kind: DaemonSet
        metadata:
          name: kube-vip-ds
        spec:
          template:
            spec:
              serviceAccountName: kube-vip
              nodeSelector:
                node-role.kubernetes.io/worker: ""
              containers:
              - name: kube-vip
                env:
                - name: vip_interface
                  value: "ens18"
                - name: vip_subnet
                  value: "16"
                volumeMounts:
                - mountPath: /etc/kubernetes/admin.conf
                  $patch: delete
              volumes:
              - name: kubeconfig
                $patch: delete
    - target:
        kind: ConfigMap
        name: kubevip
        namespace: kube-system
      patch: |-
        - op: add
          path: /data/range-global
          value: "172.20.6.117-172.20.6.120"
    ```

4. 驗證 yaml 設定檔是否有誤

    ```
    kubectl apply -k . --dry-run=server
    ```

    正確執行結果：

    ```
    serviceaccount/kube-vip created (server dry run)
    serviceaccount/kube-vip-cloud-controller created (server dry run)
    clusterrole.rbac.authorization.k8s.io/system:kube-vip-cloud-controller-role created (server dry run)
    clusterrole.rbac.authorization.k8s.io/system:kube-vip-role created (server dry run)
    clusterrolebinding.rbac.authorization.k8s.io/system:kube-vip-binding created (server dry run)
    clusterrolebinding.rbac.authorization.k8s.io/system:kube-vip-cloud-controller-binding created (server dry run)
    configmap/kubevip created (server dry run)
    Warning: spec.template.spec.affinity.nodeAffinity.preferredDuringSchedulingIgnoredDuringExecution[1].preference.matchExpressions[0].key: node-role.kubernetes.io/master is use "node-role.kubernetes.io/control-plane" instead
    deployment.apps/kube-vip-cloud-provider created (server dry run)
    daemonset.apps/kube-vip-ds created (server dry run)
    ```

5. 部署 kube-vip On-Premises Cloud Controller

    ```
    kubectl apply -k .
    ```

6. 檢視 kube-vip On-Premises Cloud Controller pods 運作狀態

    ```
    kubectl -n kube-system get pods -l component=kube-vip-cloud-provider
    ```

    執行結果：

    ```
    NAME                                      READY   STATUS    RESTARTS   AGE
    kube-vip-cloud-provider-d9b9446b9-6ll6t   1/1     Running   0          45s
    ```

7. 檢視 kube-vip ds pods 運作狀態，並確認 pods 只跑在 worker nodes 上

    ```
    kubectl -n kube-system get pods -l app.kubernetes.io/name=kube-vip-ds -o wide
    ```

    執行結果：

    ```
    NAME                READY   STATUS    RESTARTS   AGE   IP             NODE                       NOMINATED NODE   READINESS GATES
    kube-vip-ds-kglmc   1/1     Running   0          59s   172.20.6.114   topgun-w1.kubeantony.com   <none>           <none>
    kube-vip-ds-krst5   1/1     Running   0          59s   172.20.6.115   topgun-w2.kubeantony.com   <none>           <none>
    kube-vip-ds-r8cmc   1/1     Running   0          59s   172.20.6.116   topgun-w3.kubeantony.com   <none>           <none>
    ```

### 4.3. 安裝 Ingress NGINX Controller

1. 切換工作目錄

    ```
    cd ${HOME}/k8s/addon/ingress/ingress-nginx
    ```

2. 編輯 Kustomization File

    ```
    nano kustomization.yaml
    ```

    檔案內容如下：

    ```
    apiVersion: kustomize.config.k8s.io/v1beta1
    kind: Kustomization
    resources:
    #- ./deployment.yaml
    - ./daemonSet.yaml

    images:
    - name: registry.k8s.io/ingress-nginx/controller
      newName: harbor.example.com/library/ingress-nginx/controller
      newTag: v1.13.3
    - name: registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.6.3
      newName: harbor.example.com/library/ingress-nginx/kube-webhook-certgen
      newTag: v1.6.3

    patches:
    - target:
        kind: DaemonSet
        name: ingress-nginx-controller
      patch: |-
        apiVersion: apps/v1
        kind: DaemonSet
        metadata:
          name: ingress-nginx-controller
        spec:
          template:
            spec:
              nodeSelector:
                node-role.kubernetes.io/worker: ""
    ```

3. 驗證 yaml 設定檔是否有誤

    ```
    kubectl apply -k . --dry-run=server
    ```

    正確執行結果：

    ```
    namespace/ingress-nginx created (server dry run)
    clusterrole.rbac.authorization.k8s.io/ingress-nginx created (server dry run)
    clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created (server dry run)
    clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created (server dry run)
    clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created (server dry run)
    ingressclass.networking.k8s.io/nginx created (server dry run)
    validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created (server dry run)
    Error from server (NotFound): error when creating ".": namespaces "ingress-nginx" not found
    Error from server (NotFound): error when creating ".": namespaces "ingress-nginx" not found
    Error from server (NotFound): error when creating ".": namespaces "ingress-nginx" not found
    Error from server (NotFound): error when creating ".": namespaces "ingress-nginx" not found
    Error from server (NotFound): error when creating ".": namespaces "ingress-nginx" not found
    Error from server (NotFound): error when creating ".": namespaces "ingress-nginx" not found
    Error from server (NotFound): error when creating ".": namespaces "ingress-nginx" not found
    Error from server (NotFound): error when creating ".": namespaces "ingress-nginx" not found
    Error from server (NotFound): error when creating ".": namespaces "ingress-nginx" not found
    Error from server (NotFound): error when creating ".": namespaces "ingress-nginx" not found
    Error from server (NotFound): error when creating ".": namespaces "ingress-nginx" not found
    Error from server (NotFound): error when creating ".": namespaces "ingress-nginx" not found
    ```

5. 部署 Ingress NGINX Controller

    ```
    kubectl apply -k .
    ```

6. 檢視 Ingress NGINX Controller pods 運作狀態，並確認是否只在 woker nodes 上執行

    ```
    kubectl -n ingress-nginx get pods -o wide
    ```

    執行結果：

    ```
    NAME                             READY   STATUS    RESTARTS   AGE   IP             NODE                       NOMINATED NODE   READINESS GATES
    ingress-nginx-controller-4tgxs   1/1     Running   0          90s   10.244.5.109   topgun-w2.kubeantony.com   <none>           <none>
    ingress-nginx-controller-n9t75   1/1     Running   0          67s   10.244.6.115   topgun-w3.kubeantony.com   <none>           <none>
    ingress-nginx-controller-thkd4   1/1     Running   0          44s   10.244.1.190   topgun-w1.kubeantony.com   <none>           <none>
    ```

7. 建立測試目錄

    ```
    mkdir test
    ```

9. 編輯測試用 deployment

    ```
    nano test/test-deployment
    ```

    檔案內容：

    ```
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      labels:
        app: httpd
      name: httpd
    spec:
      replicas: 3
      selector:
        matchLabels:
          app: httpd
      strategy: {}
      template:
        metadata:
          labels:
            app: httpd
        spec:
          containers:
          - image: harbor.example.com/library/library/busybox:stable
            name: busybox
            command: ["/bin/sh", "-c"]
            args:
             - echo haha > /tmp/index.html;
               httpd -p 888 -h /tmp -f
    ```

7. 編輯 K8s service yaml

    ```
    nano test/test-svc.yaml
    ```

    檔案內容：

    ```
    apiVersion: v1
    kind: Service
    metadata:
      labels:
        app: httpd
      name: test-svc
    spec:
      ports:
      - port: 80
        protocol: TCP
        targetPort: 888
      selector:
        app: httpd
    ```

8. 編輯 K8s ingress yaml

    ```
    nano test/test-ing.yaml
    ```

    檔案內容：

    ```
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: minimal-ingress
    spec:
      ingressClassName: nginx
      rules:
      - host: test.k8s.io
        http:
          paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: test-svc
                port:
                  number: 80
    ```

10. 部署測試 deployment, service 和 ingress

    ```
    kubectl apply -f test/
    ```

11. 確認 pods 狀態

    ```
    kubectl get pods
    ```

    執行結果：

    ```
    NAME                     READY   STATUS    RESTARTS   AGE
    httpd-77c9cfcf7d-2tqks   1/1     Running   0          70m
    httpd-77c9cfcf7d-4g9fl   1/1     Running   0          70m
    httpd-77c9cfcf7d-6w2l2   1/1     Running   0          70m
    ```

12. 確認 ingress 有拿到節點的 ip

    ```
    kubectl get ing
    ```

    執行結果：

    ```
    NAME              CLASS   HOSTS         ADDRESS                                  PORTS   AGE
    minimal-ingress   nginx   test.k8s.io   172.20.6.114,172.20.6.115,172.20.6.116   80      66m
    ```

13. 驗證可以存取網站服務

    ```
    for i in {114..116}; do time curl -H "Host: test.k8s.io" http://172.20.6.${i};echo ;done
    ```

    正確執行結果：

    ```
    haha

    real    0m0.006s
    user    0m0.003s
    sys     0m0.002s

    haha

    real    0m0.005s
    user    0m0.002s
    sys     0m0.002s

    haha

    real    0m0.005s
    user    0m0.002s
    sys     0m0.001s

    ```