**Follow these videos & repo:** KodeKloud - Install Kubernetes Cluster from Scratch https://www.youtube.com/playlist?list=PL2We04F3Y_41jYdadX55fdJplDvgNGENo Repository https://github.com/mmumshad/kubernetes-the-hard-way/ **My Environment** OS: Windows 11(x86) VirtualBox: 7.0.14 Vagrant: 2.4.1 # Prerequisite 1. 安裝 VirtualBox & Vagrant 2. 進到有 `Vagrantfile` 的路徑底下 ``` vagrant up ``` 如果卡在 `ssh auth method private key` 的地方，可以考慮移除 VirtualBox 重新安裝 3. 下面的步驟進行一半可以暫停 ```bash! # shut down the VMs grafefully vagrant halt ``` 如果要接續，再執行一次 `vagrant up` 即可 # Steps 啟動 VMs 後 ## 1. 操作者可以 ssh 進每一台 VM 用 `vagrant ssh <machine name>` ssh 進 VM ```bash! vagrant ssh master-1 ``` 或是直接用 ssh + private key + account@IP ```bash! ssh -i vagrant/.vagrant/machines/master-1/virtualbox/private_key vagrant@192.168.56.11 ssh -i vagrant/.vagrant/machines/master-2/virtualbox/private_key vagrant@192.168.56.12 ssh -i vagrant/.vagrant/machines/worker-1/virtualbox/private_key vagrant@192.168.56.21 ssh -i vagrant/.vagrant/machines/worker-2/virtualbox/private_key vagrant@192.168.56.22 ssh -i vagrant/.vagrant/machines/loadbalancer/virtualbox/private_key vagrant@192.168.56.30 ``` ## 2. 讓主要執行 administrative task 的 VM 可以連接到其他 VM 選擇 master-1 當主要的 VM，在 master-1 執行 ```bash ssh-keygen ``` 會在 `~/.ssh` 底下產生 `id_rsa` & `id_rsa.pub` 複製 `~/.ssh/id_rsa.pub` 到其他 4 台 VM: 把下面這個指令到另外四台 VM 中執行，key 要換成你自己的 `id_rsa.pub` ```bash! cat >> ~/.ssh/authorized_keys << EOF ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSRnJ2r7SrwgbBA3HTr3WO/R2sDo8BP4zJwdJ3HZ26fkA1p/wkq6qQw1oEIK0CpIdgb96if+jqEuypYVxO9KFJ+N/TCuty1kXv4XSglW2o4TeDtq2bD73vEXpLvYGJMmz7P4OhquofNGN8U0vVcZd8tOn3seo1OO8Sq0Yz+zeno3ryvQryu/7V5nwP4O/btexiKOpoVNyDc1iPNYy2e5oQqucMyMoxJ/JSqgYt32nYDJMLXd+meFUjkXwYO63T8iTm8TO7y3OB63afRtsfiiCqyaXCeP4hXONI8mpVjpguYQcMGVzWbJlXP32NU8SatA78i0EJp+7s7GLYp6Vj0Goodh18Ah7N0qRN05V/BYorWFrbDUr4hg+aihCXyMcUZlbL6VsISQwEVwZPX9I6qMhKfp9QvIqSRbu52GH2WBbIAyzEkDzDrOqEmCjzc511tFXyckMZyJWr4JsIK2TaBLD+nCkq2EXySsfqvH0LT+B1JEUqslo2azpUL5AmT2bW7sU= vagrant@master-1 EOF ``` 這時再去 master-1，就能從 master-1 ssh to other VMs ```bash! ssh master-2 ``` ## 3. 安裝 `kubectl` 照官網步驟安裝 ```bash! # check version kubectl version ``` ## 4. 產生各種 key, csr, & crt 照著做: https://github.com/mmumshad/kubernetes-the-hard-way/blob/master/docs/04-certificate-authority.md 首先產生 `ca.key` & `ca.csr`，然後自簽名產生 `ca.crt` 後面就會用 `ca.key` & `ca.crt` 幫每個組件或 client 要用到的 crt 簽名 包括 - * admin: cluster admin * kubelet * controller manager * kube proxy * scheduler * kube API server -> 要產生 `openssl.cnf` 記一些額外的資訊: alternative names * API server as client to Kubelet (apiserver-kubelet-client.key & apiserver-kubelet-client.crt) * ETCD server * Service Account -> for controller manager use to generate and sign service account tokens [**!not sure!**] 最後分派這些 cert & key 到 master nodes & worker nodes ```bash! { for instance in master-2; do scp -o StrictHostKeyChecking=no ca.crt ca.key \ kube-apiserver.key kube-apiserver.crt \ apiserver-kubelet-client.crt apiserver-kubelet-client.key \ service-account.key service-account.crt \ etcd-server.key etcd-server.crt \ kube-controller-manager.key kube-controller-manager.crt \ kube-scheduler.key kube-scheduler.crt \ ${instance}:~/ done for instance in worker-1 worker-2 ; do scp ca.crt kube-proxy.crt kube-proxy.key ${instance}:~/ done } ``` ## 5. kubeconfig file 照著做: https://github.com/mmumshad/kubernetes-the-hard-way/blob/master/docs/05-kubernetes-configuration-files.md 要幫 (API server's) client 建立 kubeconfig file - * 同樣在 contorlplane 的 -> 透過 127.0.0.1 跟 API server 接觸就好 * controller manager * scheduler * 不在 controlplane 的 -> 透過 Load Balancer in front of masters * kube-proxy * admin user 這兩種情形設置的 `--server=<cluster IP>` 會不同 e.g. `kube-controller-manager` - on master nodes `--server=https://127.0.0.1:6443` ```bash { kubectl config set-cluster kubernetes-the-hard-way \ --certificate-authority=/var/lib/kubernetes/pki/ca.crt \ --server=https://127.0.0.1:6443 \ --kubeconfig=kube-controller-manager.kubeconfig kubectl config set-credentials system:kube-controller-manager \ --client-certificate=/var/lib/kubernetes/pki/kube-controller-manager.crt \ --client-key=/var/lib/kubernetes/pki/kube-controller-manager.key \ --kubeconfig=kube-controller-manager.kubeconfig kubectl config set-context default \ --cluster=kubernetes-the-hard-way \ --user=system:kube-controller-manager \ --kubeconfig=kube-controller-manager.kubeconfig kubectl config use-context default --kubeconfig=kube-controller-manager.kubeconfig } ``` `kube-proxy` - on worker nodes `--server=https://${LOADBALANCER}:6443` ``` { kubectl config set-cluster kubernetes-the-hard-way \ --certificate-authority=/var/lib/kubernetes/pki/ca.crt \ --server=https://${LOADBALANCER}:6443 \ --kubeconfig=kube-proxy.kubeconfig kubectl config set-credentials system:kube-proxy \ --client-certificate=/var/lib/kubernetes/pki/kube-proxy.crt \ --client-key=/var/lib/kubernetes/pki/kube-proxy.key \ --kubeconfig=kube-proxy.kubeconfig kubectl config set-context default \ --cluster=kubernetes-the-hard-way \ --user=system:kube-proxy \ --kubeconfig=kube-proxy.kubeconfig kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig } ``` 建立完後再傳到其他 master / worker nodes ## 6. Data Encryption Keys **[not sure what's this for]** https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/#encrypting-your-data > Kubernetes stores a variety of data including cluster state, application configurations, and secrets. Kubernetes supports the ability to encrypt cluster data at rest, that is, the data stored within etcd 產生隨機字串 ``` head -c 32 /dev/urandom | base64 ``` 這個指令的含義如下 - from ChatGPT 3.5 > `head -c 32`: 這個部分使用 head 命令，-c 選項表示要顯示的字節數（bytes），這裡是 32 個字節。所以這個部分的作用是從 /dev/urandom 中讀取 32 個隨機字節。 > `/dev/urandom`: 這是一個特殊的設備文件（device file）在 Unix-like 系統中，它提供了一個無窮流的隨機數據。/dev/urandom 通常被用來生成安全的隨機數據。 > `|`: 這是管道（pipe）操作符，它將前一個命令的輸出作為後一個命令的輸入。 > `base64`: 這個部分使用 base64 命令，它將輸入的二進制數據轉換為 base64 編碼。在這裡，它將從 /dev/urandom 中讀取的 32 個隨機字節轉換為 base64 編碼的字符串。 > 總的來說，這個指令的目的是生成一個包含 32 字節隨機數據的 base64 編碼字符串。這種生成方式通常用於生成密鑰、令牌等需要高度隨機性的數據。 產生 `encryption-config.yaml` 並分布到其他 master ## 7. 安裝 ETCD 目前設計是 ETCD & control plane 在同一台 需要在所有 master node 都裝 ETCD 照著做: https://github.com/mmumshad/kubernetes-the-hard-way/blob/master/docs/07-bootstrapping-etcd.md **Summary of steps:** * 下載 etcd binaries * extract 後安裝(移到 `/usr/local/bin/`) * configure * 建立相對應的 dir * 複製 cert & key 到該放的地方 * 設置權限 * 建立 `etcd.service` systemd unit file * 裡面的路徑、IP 要寫對 * 啟動 etcd server * `sudo systemctl daemon-reload` - reload systemd manager configuration. 當 systemd units 設定檔有變化時(e.g. .service)，跑這個指令讓 systemd 讀的最新的設定檔 * `sudo systemctl enable etcd` - 讓 `etcd` service 在系統重啟時自動啟動 * `sudo systemctl start etcd` - start etcd service <驗證> ``` sudo ETCDCTL_API=3 etcdctl member list \ --endpoints=https://127.0.0.1:2379 \ --cacert=/etc/etcd/ca.crt \ --cert=/etc/etcd/etcd-server.crt \ --key=/etc/etcd/etcd-server.key ``` ## 8. 安裝 Control Plane 元件 https://github.com/mmumshad/kubernetes-the-hard-way/blob/master/docs/08-bootstrapping-kubernetes-controllers.md 安裝以下元件，與安裝 etcd server 步驟類似，下載 binaries, make it executable, move it to `/usr/local/bin`, configure, 建立 `.service` file * kube-apiserver * kube-controller-manager * kube-scheduler (kubectl -> installed in Step 3) <驗證> - `kubectl get componentstatuses` -> got `Warning: v1 ComponentStatus is deprecated in v1.19+` - Use below to verify ``` curl -k https://localhost:6443/livez?verbose ``` See: https://kubernetes.io/docs/reference/using-api/health-checks/ ## 8.1 設置 Load Balancer 這邊使用 HA Proxy，進到 Load Balancer VM 安裝 https://github.com/mmumshad/kubernetes-the-hard-way/blob/master/docs/08-bootstrapping-kubernetes-controllers.md#the-kubernetes-frontend-load-balancer ## 9. Install Container Runtime > Since v1.24 the use of dockershim has been fully deprecated and removed from the code base. [containerd replaces docker](https://kodekloud.com/blog/kubernetes-removed-docker-what-happens-now/) as the container runtime for Kubernetes, and it requires support from [CNI Plugins](https://github.com/containernetworking/plugins) to configure container networks, and [runc](https://github.com/opencontainers/runc) to actually do the job of running containers. Reference: https://github.com/containerd/containerd/blob/main/docs/getting-started.md Install containerd on worker nodes Client tool - crictl (KodeKloud 這篇沒安裝) ## 10. Bootstraping Worker Nodes - manage certs manually 根據影片這邊會分兩個 worker: * 一個手動處理 certs * 一個讓 worker 自己處理 certs，之後更新 certs 讓 worker 自己更新 **Summary of steps:** 在 `master-1` 執行 * 建立 key & cert for `worker-1` * 建立 kubeconfig for `worker-1` kubelet * set Load Balancer IP as server IP (`--server=https://${LOADBALANCER}:6443`) * 把 key, cert, kubeconfig 傳給 `worker-1` 在 `worker-1` 執行 * 下載 binaries (kube-proxy & kubelet) * 建立目錄 * let binaries be executable & move them to `/usr/local/bin` * configure kubelet * 移動 key, cert, kubeconfig 到特定位置 * 指定 cluster DNS 位置，通常是 `SERVICE_CIDR` 範圍中的 .10，`kubelet-config.yaml` 會用到 * 建立 `/var/lib/kubelet/kubelet-config.yaml` * 建立 `/etc/systemd/system/kubelet.service` * configure kube-proxy * 移動 kubeconfig 到特定位置 * 建立 `/var/lib/kube-proxy/kube-proxy-config.yaml`，`POD_CIDR` 要跟前面設定的一樣 * 建立 `/etc/systemd/system/kube-proxy.service` * 啟動 worker services <驗證> * 回到 `master-1`，執行 ``` kubectl get nodes --kubeconfig admin.kubeconfig ``` NotReady 因 pod networking 還沒安裝 ``` NAME STATUS ROLES AGE VERSION worker-1 NotReady <none> 24m v1.29.2 ``` ## 11. TLS Bootstrapping Worker Nodes [YouTube Video - Install Kubernetes from Scratch [13] - TLS Bootstrap Kubelet](https://youtu.be/qnD95QkphfU?si=4nvZhZonYI2l9iPH) ![image](https://hackmd.io/_uploads/Hk5ArTrna.png) **[流程待整理]** **Pre-requisite** 檢查 `kube-apiserver` 啟動指令是否包含 `--enable-bootstrap-token-auth=true`: ```bash! vagrant@master-1:/etc/systemd/system$ cat kube-apiserver.service [Unit] Description=Kubernetes API Server Documentation=https://github.com/kubernetes/kubernetes [Service] ExecStart=/usr/local/bin/kube-apiserver \ --advertise-address=192.168.56.11 \ --allow-privileged=true \ --apiserver-count=2 \ --audit-log-maxage=30 \ --audit-log-maxbackup=3 \ --audit-log-maxsize=100 \ --audit-log-path=/var/log/audit.log \ --authorization-mode=Node,RBAC \ --bind-address=0.0.0.0 \ --client-ca-file=/var/lib/kubernetes/pki/ca.crt \ --enable-admission-plugins=NodeRestriction,ServiceAccount \ --enable-bootstrap-token-auth=true \ # this ... ``` 確認 `kube-controller-manager` 指令有指定 ca cert & key，因 csr 會由 kube-controller-manager 簽署 ```bash! # in /etc/systemd/system/kube-controller-manager.service --cluster-signing-cert-file=/var/lib/kubernetes/ca.crt \ --cluster-signing-key-file=/var/lib/kubernetes/ca.key ``` **Summary of steps** 在 `master-1` 執行 * 在 `kube-system` namespace 建立 bootstrap token 的 secret * 建立 ClusterRoleBinding，把 ClusterRole `system:node-bootstrapper` 與 Group `system:bootstrappers` 連接起來 * 建立 ClusterRoleBinding，把 ClusterRole `system:certificates.k8s.io:certificatesigningrequests:nodeclient` 與 Group `system:bootstrappers` 連接起來 * 建立 ClusterRoleBinding，把 ClusterRole `system:certificates.k8s.io:certificatesigningrequests:nodeclient` 與 Group `system:nodes` 連接起來 再來到 `worker-2` 執行 * 下載 binaries (kube-proxy & kubelet) * 建立目錄 * let binaries be executable & move them to `/usr/local/bin` * 移動 key, cert 到特定位置 * configure kubelet to TLS bootstrap * 需要 Load Balancer IP, Pod CIDR, Service CIDR, Cluster DNS IP * 建立 bootstrap-kubeconfig * 建立 `/var/lib/kubelet/kubelet-config.yaml` * 這邊不用指定 tlsCertFile and tlsPrivateKeyFile * 建立 `/etc/systemd/system/kubelet.service` * **bootstrap-kubeconfig**: Location of the bootstrap-kubeconfig file. * **cert-dir**: The directory where the generated certificates are stored. * **kubeconfig**: We specify a location for this but we have not yet created it. Kubelet will create one itself upon successful bootstrap. * configure kube-proxy * 移動 kubeconfig 到特定位置 * 建立 `/var/lib/kube-proxy/kube-proxy-config.yaml`，`POD_CIDR` 要跟前面設定的一樣 * 建立 `/etc/systemd/system/kube-proxy.service` * 啟動 worker services 最後回到 `master-1` 要核准 CSR * 先觀察有哪些 CSR ``` kubectl get csr --kubeconfig admin.kubeconfig ``` Output ```bash NAME AGE SIGNERNAME REQUESTOR REQUESTEDDURATION CONDITION csr-b5s4j 38s kubernetes.io/kube-apiserver-client-kubelet system:bootstrap:07401b <none> Approved,Issued csr-cpbxk 35s kubernetes.io/kubelet-serving system:node:worker-2 <none> Pending ``` 可以看到 bootstrap 那個(client)已經被 approved * Approve server csr ```bash kubectl certificate approve --kubeconfig admin.kubeconfig csr-cpbxk ``` or ```bash! kubectl certificate approve --kubeconfig admin.kubeconfig \ $(kubectl get csr --kubeconfig admin.kubeconfig -o json | jq -r \ '.items | .[] | select(.spec.username == "system:node:worker-2") | .metadata.name') ``` <驗證> 在 `master-1` 執行 ``` kubectl get nodes --kubeconfig admin.kubeconfig ``` **? 以下 resources 是什麼時候被建立的 ?** - ClusterRole `system:node-bootstrapper` - ClusterRole `system:certificates.k8s.io:certificatesigningrequests:nodeclient` - Group `system:bootstrappers` - Group `system:nodes` See: https://kubernetes.io/docs/reference/access-authn-authz/kubelet-tls-bootstrapping/ ## 12. Configure kubectl 建立 `~/.kube/config` ```bash LOADBALANCER=$(dig +short loadbalancer) { kubectl config set-cluster kubernetes-the-hard-way \ --certificate-authority=ca.crt \ --embed-certs=true \ --server=https://${LOADBALANCER}:6443 kubectl config set-credentials admin \ --client-certificate=admin.crt \ --client-key=admin.key kubectl config set-context kubernetes-the-hard-way \ --cluster=kubernetes-the-hard-way \ --user=admin kubectl config use-context kubernetes-the-hard-way } ``` Output ```bash! vagrant@master-1:~$ cat .kube/config apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN0ekNDQVo4Q0ZFSjdxM1F6Y0FOUXZTVHg2bmhEcWxOODJOL0lNQTBHQ1NxR1NJYjNEUUVCQ3dVQU1CZ3gKRmpBVUJnTlZCQU1NRFV0VlFrVlNUa1ZVUlZNdFEwRXdIaGNOTWpRd01qSXlNVFl5TnpVNVdoY05Nall4TVRFNApNVFl5TnpVNVdqQVlNUll3RkFZRFZRUUREQTFMVlVKRlVrNUZWRVZUTFVOQk1JSUJJakFOQmdrcWhraUc5dzBCCkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQXNrL1g5R0EzR2dOZVd1a2dJNDloYng4UnUxZGtWYkl4eWVYZFpsVjAKZlBWRXQrYXl2TlNZN2o4ZDB1Z01wS3JsRldScUliZkZqQ0FqUnM0VjNNQ21KUUovVUg1OGtrREdCK3FIV2pSTQphZzFXcC9sYzMySmFHa2puZnZJaWlpa1pMVTN0RlYrR2ZzTVdHZktObEIxSWpyb0NIM1RWaXVlTUdwYlBTdkFLCndiSzNtdVZ4YkluOFpsb2lyLys4UGdIWjFNcU04bzNkNXVKNUgzalBNZ1ZYK0xZTU5tb1hUUHQweGFLeFh6YUgKRWQwazdpbFpVSjExZmdJSWVUc3U4cXIvTlBUQWtwdjRWRVkwM29IdmdVOU5RSmNjUllaejhTM1ZmYllBaFExUApMSG1uZnp1THQ1eUJnY1hWclhEMVNmRlBwcmVTUWc0OXl6OVo0bUlzbXVuMlNRSURBUUFCTUEwR0NTcUdTSWIzCkRRRUJDd1VBQTRJQkFRQTdKUDhWWVFXalJoOFRIQW5PaEl3UmtKVHVPQ1FaKzRWc2pQZ0ZrYWFVSTYzSUd3ZHcKaithUjc1aWEydHc3dnk4REE5c01KN0UvYWRpVVhPdGh4Z0hCd0RJODA1SkRMSVAvQStyMG5MbHZRcnRxclJiegpjbVc5ZlR5U05uTEJtNW9wMXcydjQxR3pFdGJ4QXpmUG5qWEozT3UrZDVUTVlYaE9QSUZuSitTek9XdCs3SXJoClB6SzZBdlUrNkdZeUJneWo0SElibWh6UXk2OFlRbnFEbm5LUVpMWjZoOU4rVjVZQnUwS25VYWZjTnVSS0xNeVcKMSt5WWVUUlMwYm1JZkNWSVJ5M29CWDl6Sk1yNm52bWsxNjJmL05JYzV5SmdpYnNaR2NPdzVSSkJiaG9NRjgwZwpxbjE4eDYyVkZzR0h5MFZwaE12dU1UbHB1SE1zOGErOWJjUUIKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo= server: https://192.168.56.30:6443 name: kubernetes-the-hard-way contexts: - context: cluster: kubernetes-the-hard-way user: admin name: kubernetes-the-hard-way current-context: kubernetes-the-hard-way kind: Config preferences: {} users: - name: admin user: client-certificate: /home/vagrant/admin.crt client-key: /home/vagrant/admin.key ``` ## 13. Configure Pod Networking 示範是用 Weave，但 Weave 收掉啦!!! 這邊改裝 flannel https://github.com/flannel-io/flannel#deploying-flannel-manually 依步驟安裝 ``` kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml ``` 如果 Pod CIDR 不是設定成預設的 `10.244.0.0/16`，要下載回來自己改 > (from ChatGPT) `/etc/systemd/system/kube-controller-manager.service` 的啟動參數 `--cluster-cidr` 是 Kubernetes controller manager 的啟動參數之一，它指定了 cluster 中 Pod 網路的 CIDR 範圍。 `--cluster-cidr=10.244.0.0/16` 意味著 Kubernetes cluster 將使用 IP 位址範圍從 10.244.0.0 到 10.244.255.255（CIDR 表示法為 /16）來分配給 Pod。這是一個私有的 IP 位址範圍，用於 cluster 中的 Pod IP 位址。 Pod IP 位址是在 Kubernetes cluster 中的 Pods 之間進行通信時使用的 IP 位址範圍。每個節點上的 Pod 都會從指定的 `--cluster-cidr` 中獲取其 IP 位址。 <驗證> ```bash vagrant@master-1:~$ kubectl get pods -n kube-flannel NAME READY STATUS RESTARTS AGE kube-flannel-ds-bzmln 1/1 Running 0 2m31s kube-flannel-ds-h25dg 1/1 Running 0 2m31s vagrant@master-1:~$ kubectl get all -n kube-flannel NAME READY STATUS RESTARTS AGE pod/kube-flannel-ds-bzmln 1/1 Running 0 2m37s pod/kube-flannel-ds-h25dg 1/1 Running 0 2m37s NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/kube-flannel-ds 2 2 2 2 2 <none> 2m38s ``` 可以看到會建 daemonset 另外再確認一下 nodes ```bash vagrant@master-1:~$ kubectl get nodes NAME STATUS ROLES AGE VERSION worker-1 Ready <none> 118m v1.29.2 worker-2 Ready <none> 43m v1.29.2 ``` 那到底 flannel 是怎麼作用的? --- pending to check = = ## 14. Kube API server to Kubelet Permission (skip, 照上面的步驟執行到這裡，是可以看 log 的，not sure which steps configuring the permission?) ## 15. DNS add-on 加上 DNS add-on for DNS based service discovery 使用 CoreDNS - https://coredns.io/ 這邊 Mumshad Mannambeth 已經幫忙寫好 coreDNS deploy manifest 直接 apply ``` kubectl apply -f https://raw.githubusercontent.com/mmumshad/kubernetes-the-hard-way/master/deployments/coredns.yaml ``` 如果 Service CIDR range 有改，需要更改 `coredns.yaml` 裡面的 `kube-dns` Service 的 clusterIP 位置，這會是 DNS service address。(`SERVICE_CIDR=10.96.0.0/16` -> DNS service address: `10.96.0.10`) See: https://github.com/mmumshad/kubernetes-the-hard-way/blob/master/docs/01-prerequisites.md#service-network <驗證> 在安裝 CoreDNS 之前，可以先建立一個 Pod `nginx` ``` kubectl run nginx --image=nginx:1.25.4-alpine ``` 安裝 CoreDNS 後，再建立一個 Pod `busybox` ``` kubectl run busybox --image=busybox:1.28 --restart Never --command -- sleep 1d ``` 在這兩個 Pod 中執行 `nslookup kubernetes` `busybox` ```bash vagrant@master-1:~$ kubectl exec -it busybox -- nslookup kubernetes Server: 10.96.0.10 Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local Name: kubernetes Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local ``` `nginx` ```bash vagrant@master-1:~$ kubectl exec -it nginx -- nslookup kubernetes Server: 10.96.0.10 Address: 10.96.0.10:53 ** server can't find kubernetes.cluster.local: NXDOMAIN Name: kubernetes.default.svc.cluster.local Address: 10.96.0.1 ** server can't find kubernetes.svc.cluster.local: NXDOMAIN ** server can't find kubernetes.cluster.local: NXDOMAIN ** server can't find kubernetes.svc.cluster.local: NXDOMAIN command terminated with exit code 1 ``` (Pending to check the difference. `nginx` pod can resolve `kubernetes` to `kubernetes.default.svc.cluster.local`. But there will be other not found records.) 官網寫 image `coredns/coredns` 基本上是由 scratch + CoreDNS + TLS certificates (for DoT, DoH, and gRPC) 組成，因此無法開 shell 進去看 coreDNS 的配置文件驗證。 --- **References** * Ports and Protocols: https://kubernetes.io/docs/reference/networking/ports-and-protocols/ * Kubernetes releases: https://github.com/kubernetes/kubernetes * ETCD releases: https://github.com/etcd-io/etcd (what version to be used with what version of kubernetes -> Kubernetes Release Note) * DNS for Services and Pods: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/