# rke2 v1.26.10+rke2r1 HA with Rancher * 所有 VM 皆需關防火牆 & 設定固定 ip ## 架構圖 ![image](https://hackmd.io/_uploads/SkZRjbcHp.png) ## install DNS Server * 在 sles15-sp5 安裝 DNS Server ```shell= bigred@DNS:~> sudo zypper in -t pattern dhcp_dns_server ``` * 設定 DNS Server ``` bigred@DNS:~> sudo yast dns-server ``` * 點選 Next ![image](https://hackmd.io/_uploads/By6Xkb9Bp.png) * 新增 `example.com` 域名 ![image](https://hackmd.io/_uploads/HkYIkWcrT.png) * 設定 `example.com` DNS Zones ![image](https://hackmd.io/_uploads/S1ucJZcSp.png) ![image](https://hackmd.io/_uploads/SyBRJWcBp.png) * 新增 record，分別指向 rancher 的三台機器 ![image](https://hackmd.io/_uploads/S1oHbZ9H6.png) * 設定 DNS 服務開機自動啟動 ![image](https://hackmd.io/_uploads/rkyuW-9r6.png) * 測試名稱解析功能 ``` bigred@DNS:~> host rancher.example.com rancher.example.com has address 192.168.11.116 rancher.example.com has address 192.168.11.114 rancher.example.com has address 192.168.11.115 ``` ## install rke2 cluster & Rancher * 三台 vm nameserver 位置需指向我們自建好的 DNS Server 位置 * 在 r1 測試名稱解析功能 ``` rancher@r1:~> host rancher.example.com rancher.example.com has address 192.168.11.116 rancher.example.com has address 192.168.11.114 rancher.example.com has address 192.168.11.115 ``` ## config rke2 basic parameters ### 安裝 r1 control-plane ``` rancher@r1:~> curl -sfL https://get.rke2.io --output install.sh rancher@r1:~> chmod +x install.sh rancher@r1:~> sudo mkdir -p /etc/rancher/rke2/ [sudo] root 的密碼： rancher@r1:~> sudo vim /etc/rancher/rke2/config.yaml rancher@r1:~> cat /etc/rancher/rke2/config.yaml node-name: - "r1" token: my-shared-secret etcd-extra-env: TZ=Asia/Taipei kube-apiserver-extra-env: TZ=Asia/Taipei kube-controller-manager-extra-env: TZ=Asia/Taipei kube-proxy-extra-env: TZ=Asia/Taipei kube-scheduler-extra-env: TZ=Asia/Taipei cloud-controller-manager-extra-env: TZ=Asia/Taipei ``` ``` rancher@r1:~> sudo INSTALL_RKE2_CHANNEL=v1.26.10+rke2r1 ./install.sh [WARN] /usr/local is read-only or a mount point; installing to /opt/rke2 [INFO] using v1.26.10+rke2r1 as release [INFO] downloading checksums at https://github.com/rancher/rke2/releases/download/v1.26.10+rke2r1/sha256sum-amd64.txt [INFO] downloading tarball at https://github.com/rancher/rke2/releases/download/v1.26.10+rke2r1/rke2.linux-amd64.tar.gz [INFO] verifying tarball [INFO] unpacking tarball file to /opt/rke2 [INFO] updating tarball contents to reflect install path [INFO] moving systemd units to /etc/systemd/system [INFO] install complete; you may want to run: export PATH=$PATH:/opt/rke2/bin rancher@r1:~> export PATH=$PATH:/opt/rke2/bin ``` * enable rke2 and setup kubeconfig ```shell= rancher@r1:~> sudo systemctl enable rke2-server Created symlink /etc/systemd/system/multi-user.target.wants/rke2-server.service → /etc/systemd/system/rke2-server.service. rancher@r1:~> sudo systemctl start rke2-server rancher@r1:~> mkdir .kube rancher@r1:~> sudo cp /etc/rancher/rke2/rke2.yaml .kube/config [sudo] root 的密碼： rancher@r1:~> sudo chown rancher .kube/config rancher@r1:~> sudo cp /var/lib/rancher/rke2/bin/kubectl /usr/local/bin/ rancher@r1:~> sudo cp /opt/rke2/bin/* /usr/local/bin/ ``` ### 安裝 r2 control-plane ``` rancher@r2:~> curl -sfL https://get.rke2.io --output install.sh rancher@r2:~> chmod +x install.sh rancher@r2:~> sudo mkdir -p /etc/rancher/rke2/ [sudo] root 的密碼： rancher@r2:~> sudo vim /etc/rancher/rke2/config.yaml ``` * confing rke2 ha control plan ``` # 做 rke2 ha 要指定第一台 master node 的 ip，port 是 9345 rancher@r2:~> cat /etc/rancher/rke2/config.yaml server: https://192.168.11.114:9345 node-name: - "r2" token: my-shared-secret etcd-extra-env: TZ=Asia/Taipei kube-apiserver-extra-env: TZ=Asia/Taipei kube-controller-manager-extra-env: TZ=Asia/Taipei kube-proxy-extra-env: TZ=Asia/Taipei kube-scheduler-extra-env: TZ=Asia/Taipei cloud-controller-manager-extra-env: TZ=Asia/Taipei ``` ``` rancher@r2:~> sudo INSTALL_RKE2_CHANNEL=v1.26.10+rke2r1 ./install.sh [WARN] /usr/local is read-only or a mount point; installing to /opt/rke2 [INFO] using v1.26.10+rke2r1 as release [INFO] downloading checksums at https://github.com/rancher/rke2/releases/download/v1.26.10+rke2r1/sha256sum-amd64.txt [INFO] downloading tarball at https://github.com/rancher/rke2/releases/download/v1.26.10+rke2r1/rke2.linux-amd64.tar.gz [INFO] verifying tarball [INFO] unpacking tarball file to /opt/rke2 [INFO] updating tarball contents to reflect install path [INFO] moving systemd units to /etc/systemd/system [INFO] install complete; you may want to run: export PATH=$PATH:/opt/rke2/bin rancher@r2:~> export PATH=$PATH:/opt/rke2/bin ``` * enable rke2 and setup kubeconfig ``` rancher@r2:~> sudo systemctl enable rke2-server Created symlink /etc/systemd/system/multi-user.target.wants/rke2-server.service → /etc/systemd/system/rke2-server.service. rancher@r2:~> sudo systemctl start rke2-server rancher@r2:~> sudo cp /opt/rke2/bin/* /usr/local/bin/ ``` ### 安裝 r3 control-plane ``` rancher@r3:~> curl -sfL https://get.rke2.io --output install.sh rancher@r3:~> chmod +x install.sh rancher@r3:~> sudo mkdir -p /etc/rancher/rke2/ [sudo] root 的密碼： rancher@r3:~> sudo vim /etc/rancher/rke2/config.yaml ``` * confing rke2 ha control plan ``` # 做 rke2 ha 要指定第一台 master node 的 ip，port 是 9345 rancher@r3:~> cat /etc/rancher/rke2/config.yaml server: https://192.168.11.114:9345 node-name: - "r3" token: my-shared-secret etcd-extra-env: TZ=Asia/Taipei kube-apiserver-extra-env: TZ=Asia/Taipei kube-controller-manager-extra-env: TZ=Asia/Taipei kube-proxy-extra-env: TZ=Asia/Taipei kube-scheduler-extra-env: TZ=Asia/Taipei cloud-controller-manager-extra-env: TZ=Asia/Taipei ``` ``` rancher@r3:~> sudo INSTALL_RKE2_CHANNEL=v1.26.10+rke2r1 ./install.sh [WARN] /usr/local is read-only or a mount point; installing to /opt/rke2 [INFO] using v1.26.10+rke2r1 as release [INFO] downloading checksums at https://github.com/rancher/rke2/releases/download/v1.26.10+rke2r1/sha256sum-amd64.txt [INFO] downloading tarball at https://github.com/rancher/rke2/releases/download/v1.26.10+rke2r1/rke2.linux-amd64.tar.gz [INFO] verifying tarball [INFO] unpacking tarball file to /opt/rke2 [INFO] updating tarball contents to reflect install path [INFO] moving systemd units to /etc/systemd/system [INFO] install complete; you may want to run: export PATH=$PATH:/opt/rke2/bin rancher@r3:~> export PATH=$PATH:/opt/rke2/bin ``` * enable rke2 and setup kubeconfig ``` rancher@r3:~> sudo systemctl enable rke2-server Created symlink /etc/systemd/system/multi-user.target.wants/rke2-server.service → /etc/systemd/system/rke2-server.service. rancher@r3:~> sudo systemctl start rke2-server rancher@r3:~> sudo cp /opt/rke2/bin/* /usr/local/bin/ ``` * 回到第一台 master 檢查 rke2 cluster 狀態 ```! rancher@r1:~> kubectl get nodes NAME STATUS ROLES AGE VERSION r1 Ready control-plane,etcd,master 36m v1.26.10+rke2r1 r2 Ready control-plane,etcd,master 29m v1.26.10+rke2r1 r3 Ready control-plane,etcd,master 24m v1.26.10+rke2r1 ``` ## install helm3 ```shell=! rancher@r1:~> wget https://get.helm.sh/helm-v3.8.2-linux-amd64.tar.gz --2022-09-21 09:06:57-- https://get.helm.sh/helm-v3.8.2-linux-amd64.tar.gz Resolving get.helm.sh (get.helm.sh)... 152.199.39.108, 2606:2800:247:1cb7:261b:1f9c:2074:3c Connecting to get.helm.sh (get.helm.sh)|152.199.39.108|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 13633605 (13M) [application/x-tar] Saving to: ‘helm-v3.8.2-linux-amd64.tar.gz’ helm-v3.8.2-linux-amd64.tar.gz 100%[=============================================================>] 13.00M 5.95MB/s in 2.2s 2022-09-21 09:07:00 (5.95 MB/s) - ‘helm-v3.8.2-linux-amd64.tar.gz’ saved [13633605/13633605] rancher@r1:~> tar zxvf helm-v3.8.2-linux-amd64.tar.gz linux-amd64/ linux-amd64/helm linux-amd64/LICENSE linux-amd64/README.md rancher@r1:~> ls bin helm-v3.8.2-linux-amd64.tar.gz install.sh linux-amd64 public_html rancher@r1:~> sudo cp linux-amd64/helm /usr/local/bin/ rancher@r1:~> helm --help The Kubernetes package manager Common actions for Helm: - helm search: search for charts - helm pull: download a chart to your local directory to view - helm install: upload the chart to Kubernetes - helm list: list releases of charts ... ... ... ``` ## install rancher and cert-manager * 新增 cert-manager & rancher helm repo ```shell=! rancher@r1:~> helm repo add rancher-stable https://releases.rancher.com/server-charts/stable "rancher-stable" has been added to your repositories rancher@r1:~> helm repo add jetstack https://charts.jetstack.io "jetstack" has been added to your repositories rancher@rms1:~> helm repo update ...Successfully got an update from the "jetstack" chart repository ...Successfully got an update from the "rancher-stable" chart repository Update Complete. ⎈Happy Helming!⎈ ``` * 安裝 cert-manager v1.11.0 ``` rancher@r1:~> kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.11.0/cert-manager.crds.yaml customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created rancher@r1:~> helm install cert-manager jetstack/cert-manager \ --namespace cert-manager \ --create-namespace \ --version v1.11.0 NAME: cert-manager LAST DEPLOYED: Wed Sep 21 09:11:15 2022 NAMESPACE: cert-manager STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: cert-manager v1.11.0 has been deployed successfully! In order to begin issuing certificates, you will need to set up a ClusterIssuer or Issuer resource (for example, by creating a 'letsencrypt-staging' issuer). More information on the different types of issuers and how to configure them can be found in our documentation: https://cert-manager.io/docs/configuration/ For information on how to configure cert-manager to automatically provision Certificates for Ingress resources, take a look at the `ingress-shim` documentation: https://cert-manager.io/docs/usage/ingress/ rancher@r1:~> kubectl get pods --namespace cert-manager NAME READY STATUS RESTARTS AGE cert-manager-76d44b459c-zhpp2 1/1 Running 0 32s cert-manager-cainjector-9b679cc6-6tzd8 1/1 Running 0 32s cert-manager-webhook-57c994b6b9-4dfvs 1/1 Running 0 32s ``` * 安裝 rancher v2.7.9 * hostname 取為 `rancher.example.com` ```! # 可以先搜尋要安裝什麼版本 rancher@r1:~> helm search repo rancher-stable --versions rancher@r1:~> kubectl create namespace cattle-system namespace/cattle-system created rancher@r1:~> helm install rancher rancher-stable/rancher --namespace cattle-system --set hostname=rancher.example.com --version 2.7.9 --set global.cattle.psp.enabled=false NAME: rancher LAST DEPLOYED: Wed Sep 21 09:14:06 2022 NAMESPACE: cattle-system STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: Rancher Server has been installed. NOTE: Rancher may take several minutes to fully initialize. Please standby while Certificates are being issued, Containers are started and the Ingress rule comes up. Check out our docs at https://rancher.com/docs/ If you provided your own bootstrap password during installation, browse to https://rancher.example.com to get started. If this is the first time you installed Rancher, get started by running this command and clicking the URL it generates: echo https://rancher.example.com/dashboard/?setup=$(kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{.data.bootstrapPassword|base64decode}}') To get just the bootstrap password on its own, run: kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{.data.bootstrapPassword|base64decode}}{{ "\n" }}' Happy Containering! ``` ## check rancher status ```shell= rancher@r1:~> kubectl -n cattle-system get po NAME READY STATUS RESTARTS AGE rancher-7fd65d9cd6-8krrq 0/1 ContainerCreating 0 16s rancher-7fd65d9cd6-h28fw 0/1 ContainerCreating 0 16s rancher-7fd65d9cd6-k9hrr 0/1 ContainerCreating 0 16s rancher@r1:~> watch kubectl -n cattle-system get po rancher@r1:~> kubectl -n cattle-system rollout status deploy/rancher Waiting for deployment "rancher" rollout to finish: 0 of 3 updated replicas are available... Waiting for deployment spec update to be observed... Waiting for deployment "rancher" rollout to finish: 0 of 3 updated replicas are available... Waiting for deployment "rancher" rollout to finish: 1 of 3 updated replicas are available... Waiting for deployment "rancher" rollout to finish: 2 of 3 updated replicas are available... deployment "rancher" successfully rolled out rancher@r1:~> kubectl -n cattle-system get po NAME READY STATUS RESTARTS AGE rancher-7fd65d9cd6-8krrq 1/1 Running 1 (51s ago) 3r11s rancher-7fd65d9cd6-h28fw 1/1 Running 0 3r11s rancher-7fd65d9cd6-k9hrr 1/1 Running 1 (51s ago) 3r11s ``` ## 檢查名稱解析 ```! rancher@r1:~> curl -k -H "host: rancher.example.com" https://192.168.11.116/dashboard/ ``` * 後續進入 rancher UI 需透過 ingress ，因此需要透過解析 `rancher.example.com` 登入，如果要繞過 ingress 可以再開一個 nodeport 服務 ``` rancher@r1:~> kubectl -n cattle-system get ing NAME CLASS HOSTS ADDRESS PORTS AGE rancher <none> rancher.example.com 192.168.11.114,192.168.11.115,192.168.11.116 80, 443 6d9h ``` ## 登入 rancher ###### tags: `work`