IMRC - Kubernetes v1.21 Instalation Guide
with Ubuntu 20.04
===
###### tags: `IMRC` `Container` `Docker` `Kubernetes` `v1.21` `Hybrid Cluster` `Mixed Nodes` `Linux` `Windows` `Documentation`
> Written by Benny Suryajaya, compiled from various sources.
> NOTE: This guide is far from perfect. Please contribute to fix and/or complete this guide.
## Preface
A continuation of the previous [Kubernetes v1.18 Installation Guide](https://hackmd.io/@kolmibeni/k8s-118-install).
Since it involves newer versions of environment (Kubernetes v1.21 and Ubuntu 20.04), installation steps may differ from the previous guide.
This guide is a perpetual work-in-progress. Updates, changes, and improvements may be added in the future.
## Table of Content
[ToC]
## 1. Install Docker in Linux
Prepare node：
```bash=1
# Set your hostname
# Change into your preferred hostname
sudo hostnamectl set-hostname [your-hostname]
# Add root password
sudo passwd root
sudo sed -i '/cdrom:/ s/^\(.*\)$/#\1/g' /etc/apt/sources.list
sudo apt-get update
sudo apt-get install -y openssh-server
sudo apt-get install -y vim
sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl
sudo apt-get install -y software-properties-common gnupg2
```
Add Docker official GPG key：
```bash=+
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key --keyring /etc/apt/trusted.gpg.d/docker.gpg add -
```
Add Docker apt repository：
```bash=+
sudo add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
stable"
```
Install Docker CE, you can adjust the version here：
```bash=+
sudo apt-get update && sudo apt-get install -y \
containerd.io=1.2.13-2 \
docker-ce=5:19.03.11~3-0~ubuntu-$(lsb_release -cs) \
docker-ce-cli=5:19.03.11~3-0~ubuntu-$(lsb_release -cs)
```
Set Docker to use systemd as control group：
```bash=+
cat <<EOF | sudo tee /etc/docker/daemon.json
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
EOF
sudo mkdir -p /etc/systemd/system/docker.service.d
sudo systemctl daemon-reload
sudo systemctl restart docker
```
Give administrative right to user Docker：
```bash=+
sudo usermod -aG docker ${USER}
su ${USER}
id -nG
```
Test Docker installation：
```bash=+
docker version
```
## 2. Install Kubernetes v1.21 in Linux
Disable swap：
```bash=1
sudo swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
```
Check the content of resolv.conf:
```bash=+
cat /etc/resolv.conf
```
Modify the nameserver in resolv.conf:
```bash=+
# If your current nameserver is 127.0.0.53 (Ubuntu 18 and Ubuntu 20)
sudo sed -i -e \
's/nameserver 127.0.0.53/nameserver 8.8.8.8 8.8.4.4/i' \
/etc/resolv.conf
```
Switch to the root user
```bash=+
su
```
Update and upgrade package list:
```bash=+
sudo apt-get update -y && apt-get upgrade -y
```
Ensure net.bridge.bridge-nf-call-iptables is set to 1 in your sysctl config:
```bash=+
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sudo sysctl --system
```
Make sure that the br_netfilter module is loaded:
```bash=+
sudo modprobe br_netfilter
```
Update the `apt` package index and install packages needed to use the Kubernetes `apt` repository:
```bash=+
# Run as root user
apt-get update && sudo apt-get install -y apt-transport-https curl
```
Download the Google Cloud public signing key:
```bash=+
# Run as root user
curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
# NOTE 20230824:
# Run this too to add another pubkey, sometimes above key is invalid
curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://dl.k8s.io/apt/doc/apt-key.gpg
```
Add the Kubernetes `apt` repository:
```bash=+
# # Run as root user
# echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] \
# https://apt.kubernetes.io/ kubernetes-xenial main" \
# | sudo tee /etc/apt/sources.list.d/kubernetes.list
echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
```
Proceed to update `apt` package index, install kubelet, kubeadm and kubectl, and pin their version:
```bash=+
apt-get update -y
apt-get install -y kubelet=1.21.14-00 kubeadm=1.21.14-00 kubectl=1.21.14-00
apt-mark hold kubelet kubeadm kubectl
```
## 3. Create Kubernetes Cluster Master
> If you are using iMRC K8s Ubuntu Template (20220526), then continue from here.
### Initialize the Kubernetes cluster in Linux
Reset config, disable swap, and reset iptables：
```bash=1
# Run this as root user
sudo rm -Rf $HOME/.kube/config
sudo yes | kubeadm reset
sudo iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X
sudo swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
```
Initialize cluster:
```bash=+
# Run this as root user
kubeadm init --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/12
# OPTIONAL
# Run this command if you need to add extra IP for the API Server.
# Otherwise, skip to the next section.
kubeadm init --pod-network-cidr=10.244.0.0/16 \
--service-cidr=10.96.0.0/12 \
--apiserver-cert-extra-sans=140.116.86.178
```
Setup config file for kubectl
```bash=+
# Run this as root user
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
export KUBECONFIG=/etc/kubernetes/admin.conf
#Exit the root user
exit
# Run the commands one more time as normal user
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
```
### Setup the Network for Hybrid Cluster
Prepare Kubernetes control plane for Flannel：
```bash=1
sudo sysctl net.bridge.bridge-nf-call-iptables=1
```
~~Download & configure latest Flannel manifest for Linux:~~
```bash=+
# DOES NOT WORK, READ BELOW
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
```
Newer version of Flannel (v0.19.1) does not work with Kubernetes v1.21.
Get the working Flannel manifest (v0.18.1) from [here](https://hackmd.io/@kolmibeni/flannel-0181) and upload it to the Master node.
Modify the net-conf.json section of the flannel manifest (~/kube-flannel.yml) in order to set the VNI to 4096 and the Port to 4789. It should look as follows:
```json=
net-conf.json: |
{
"Network": "10.244.0.0/16",
"Backend": {
"Type": "vxlan",
"VNI" : 4096,
"Port": 4789
}
}
```
Apply the Flannel manifest and validate:
```bash=3
kubectl apply -f kube-flannel.yml
```
Check all Kubernetes system pods:
```bash=+
kubectl get pods -A
```
Make sure all system pods are ready and running.
If you are using Flannel 0.19, notice that Flannel pods are located in `kube-flannel` namespace, while other system pods are in `kube-system`.
If you are using Flannel 0.18, all pods are in the `kube-system` namespace.
Add Windows Flannel and kube-proxy DaemonSets:
```bash=+
curl -L https://github.com/kubernetes-sigs/sig-windows-tools/releases/latest/download/kube-proxy.yml \
| sed 's/VERSION/v1.21.14/g' \
| kubectl apply -f -
# 1. If using overlay/vxlan
kubectl apply -f https://github.com/kubernetes-sigs/sig-windows-tools/releases/latest/download/flannel-overlay.yml
# 2. If using overlay/vxlan, and network name is not 'Ethernet'
# (e.g.: 'Ethernet0 2')
curl -L https://github.com/kubernetes-sigs/sig-windows-tools/releases/latest/download/flannel-overlay.yml \
| sed 's/Ethernet/Ethernet0 2/g' \
| kubectl apply -f -
```
## 4. Install Docker in Windows
Ensure the following things in your Windows node:
* Using the English version of Windows Server 2019 version 1809
* OS Build is 17763.379 (KB4489899) or newer
* Selecting "English (United State)" as current system locale in Administrative language settings
Enable Hyper-V from PowerShell:
```shell=1
Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V -All
```
After Hyper-V has been enabled, **restart** the Windows node.
(Optional): Allow downloaded script files to run in the current session
```
Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Force -Scope Process;
```
### 1. ~~ONLINE MODE~~
> **20230913 Update**: Below steps are already deprecated since the Powershell gallery fails.
> Please go to step 3.USING INSTALLATION SCRIPT
Open an elevated PowerShell session and install the Docker-Microsoft PackageManagement Provider from the PowerShell Gallery.
```shell=+
Install-Module -Name DockerMsftProvider -Repository PSGallery -Force
```
If you're prompted to install the NuGet provider, type Y to install it as well.
If you get an error opening the PowerShell gallery, you may need to set the TLS version used by the PowerShell client to TLS 1.2. To do this, run the following command:
```shell=+
# Set the TLS version used by the PowerShell client to TLS 1.2.
[System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]::Tls12;
```
Use the PackageManagement PowerShell module to install the latest version of Docker.
```shell=+
# Install latest version
Install-Package -Name docker -ProviderName DockerMsftProvider
# Install version 19.03.11
Install-Package -Name docker -ProviderName DockerMsftProvider -RequiredVersion 19.03.11 -Force -Verbose
```
When PowerShell asks you whether to trust the package source 'DockerDefault', type A to continue the installation.
After the installation completes, restart the computer.
```shell=+
Restart-Computer -Force
```
(Obsolete) Run the install script in the machine with internet enabled.
```shell=+
.\install.ps1
```
After Docker has been installed, **restart** the Windows node.
### 2. ~~OFFLINE MODE~~
> **20230913 Update**: Below steps are already deprecated since the Powershell gallery fails.
> Please go to step 3.USING INSTALLATION SCRIPT
Run the install.ps1 with the DownloadOnly parameter in the machine with internet enabled.
This doesn’t install anything, it only downloads the zip file.
```shell=+
.\install.ps1 -DownloadOnly
```
Copy the install.ps1 file and the installation zip file over to the air-gapped (offline) machine and run the install with the -Offline parameter.
```shell=+
.\install.ps1 -Offline
```
### 3. USING INSTALLATION SCRIPT
Download the installation script for Docker Community Edition and run the script using PowerShell with elevated permission (administrator mode).
```shell=+
Invoke-WebRequest -UseBasicParsing "https://raw.githubusercontent.com/microsoft/Windows-Containers/Main/helpful_tools/Install-DockerCE/install-docker-ce.ps1" -o install-docker-ce.ps1
.\install-docker-ce.ps1
```
## 5. Install Kubernetes in Windows
> If you are using iMRC K8s Ubuntu Template (20220526), then continue from here.
Prepare installation directory (e.g.: C:\k).
```shell=
mkdir C:\k;
cd C:\k;
```
Install wins, kubelet, and kubeadm.
```shell=+
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
curl.exe -LO https://github.com/kubernetes-sigs/sig-windows-tools/releases/latest/download/PrepareNode.ps1
.\PrepareNode.ps1 -KubernetesVersion v1.21.14
```
Install kubectl
```shell=+
curl.exe -LO https://storage.googleapis.com/kubernetes-release/release/v1.18.0/bin/windows/amd64/kubectl.exe
curl.exe -LO https://storage.googleapis.com/kubernetes-release/release/v1.21.14/bin/windows/amd64/kubectl.exe
```
Add kubectl to path
```bash=+
# Add C:\k\ directory to path
[Environment]::SetEnvironmentVariable("Path", $env:Path + ";C:\k", [EnvironmentVariableTarget]::Machine)
# Add KUBECONFIG environment variable referring to C:\k\config file
[Environment]::SetEnvironmentVariable("KUBECONFIG", "C:\k\config",[EnvironmentVariableTarget]::User)
```
Using SCP, copy cluster config file from master node (**Step 3**: $HOME/.kube/config directory in Linux master node) to the C:\k directory in Windows worker node.
First, install the Posh-SSH module to the PowerShell
```shell=+
Install-Module -Name Posh-SSH
```
Remove all SSH trusted hosts
```bash=+
Get-SSHTrustedHost | Remove-SSHTrustedHost
```
Copy the config file with the following command. When asked, supply with username and password for Linux master node authentication.
```bash=+
# Replace [MASTER_NODE_IP] with the IP address of your Linux master node
# Replace [USERNAME] with the username of your Linux master node
Get-SCPFile -ComputerName '[MASTER_NODE_IP]' `
-RemoteFile "/home/[USERNAME]/.kube/config" `
-LocalFile "C:\k\config"
# OR
scp [USERNAME]@[MASTER_NODE_IP]:/home/[USERNAME]/.kube/config C:\k\config
```
Open **a new PowerShell window**, then test the kubectl functionality:
```bash=+
kubectl cluster-info
```
## 6. Join Windows Node to Cluster
Get the join token by executing the following command on the Linux master node:
```bash=1
kubeadm token create --print-join-command
```
Run the join command in the Windows worker node
```bash=+
# Example:
kubeadm join 192.168.13.43:6443 \
--token b29c63.aqvsg0953edz2ozw \
--discovery-token-ca-cert-hash \
sha256:529c64ad705bf356a2efa3c1bdb8181b852e2bc5d46f5d161dee1105e872bae6
```
Verify your installation
```shell=+
kubectl get nodes -o wide
```
All Flannel installation must be ready.
It will take approx. 15 minutes for the system to finish deploying kube-flannel and kube-proxy in Windows node.
```bash=+
# to view all system pods
kubectl get pods -n kube-system
# to view only Flannel pods
kubectl get pods -n kube-system -l app=flannel
```
## 7. Join Linux Node to Cluster
Follow **Step 1** and **Step 2** to install Docker and Kubernetes in Linux worker node.
After that, follow the commands below to join Linux worker node to the cluster.
Get the join token by executing the following command on the Linux master node:
```bash=1
kubeadm token create --print-join-command
```
Run the join command in the Linux worker node
```bash=+
# Run this command using root user:
kubeadm join 192.168.13.43:6443 \
--token b29c63.aqvsg0953edz2ozw \
--discovery-token-ca-cert-hash \
sha256:529c64ad705bf356a2efa3c1bdb8181b852e2bc5d46f5d161dee1105e872bae6
```
Copy the config file from Linux master node
```bash=+
# Replace [MASTER_NODE_IP] with the IP address of your Linux master node
# Replace [USERNAME] with the username of your Linux master node
scp [USERNAME]@[MASTER_NODE_IP]:/home/[USERNAME]/.kube/config \
/home/[USERNAME]/.kube/config
```
Set the `KUBECONFIG` variable:
```bash=+
export KUBECONFIG=$HOME/.kube/config
```
Verify your installation, then test the kubectl functionality:
```bash=+
kubectl cluster-info
```
## 8. Install Kubernetes Dashboard
In Linux master node, deploy Kubernetes Dashboard by executing this command:
```bash=1
# latest version
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.5.0/aio/deploy/recommended.yaml
```
In Linux master node, create admin-user service account to give permission for Kubernetes Dashboard to manage the cluster by executing the following commands:
```bash=+
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
name: admin-user
namespace: kubernetes-dashboard
EOF
cat <<EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: admin-user
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: admin-user
namespace: kubernetes-dashboard
EOF
```
In Linux master node, get the token for the Kubernetes Dashboard admin-user by executing the following command. This token will be used for logging in to Kubernetes Dashboard.
```bash=+
kubectl -n kubernetes-dashboard describe secret \
$(kubectl -n kubernetes-dashboard get secret \
| grep admin-user \
| awk '{print $1}')
```
In Windows worker node, open a PowerShell window and execute this command to open a proxy to Kubernetes Dashboard. Let this PowerShell window open and keep the command running.
```bash=+
kubectl proxy
```
In Windows worker node, access Kubernetes Dashboard from the following URL, then use the above-mentioned token to login.
> http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/
## 9. Install Helm
For better CI/CD workflow, we will use Helm to deploy our containerized applications to Kubernetes.
To install Helm in Windows, first install Chocolatey package manager.
```bash=
Set-ExecutionPolicy Bypass -Scope Process -Force;
[System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072;
iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))
```
Test your Chocolatey installation.
```bash=+
choco
# or
choco -?
```
Install Helm using Chocolatey.
```bash=+
choco install kubernetes-helm
```
Test your Helm installation.
```bash=+
helm
# or
helm ls
```
## [Optional 1] Delete and Re-join Windows Node
In some cases, our cluster installation fails due to some issue with the Windows worker node and we may need to remove the failed Windows node from cluster and rejoin it again.
First, get the name of Windows node by running this command in the Linux master node:
```bash=1
kubectl get nodes
```
On Linux master node, drain and delete the Windows node:
```bash=+
kubectl drain win-8uja2qmijvm --delete-local-data --force --ignore-daemonsets --delete-emptydir-data
kubectl delete node win-8uja2qmijvm
```
On Windows node, reset kubeadm and remove related services:
```bash=+
# Reset kubeadm
kubeadm reset -f
# Remove HNS Network
get-hnsnetwork | remove-hnsnetwork
# Remove firewall rule
remove-netfirewallrule kubelet
# Remove kubelet service without confirmation
nssm remove kubelet confirm
# Stop and remove wins service without confirmation
stop-service rancher-wins
nssm remove rancher-wins confirm
```
Following directories can be deleted on Windows node:
```
C:\etc
C:\opt
C:\run
C:\usr
C:\var
All contents of C:\k, EXCEPT: PrepareNode.ps1, config, and kubectl.exe
```
Get the join token again by executing the following command on the Linux master node:
```bash=+
kubeadm token create --print-join-command
```
Afterwards, on Windows node, re-install Kubernetes components and re-join the node (or follow **Step 7**)
```bash=+
# Install Kubernetes components
.\PrepareNode.ps1 -KubernetesVersion v1.21.14
# Join Windows node to cluster
kubeadm join 192.168.13.43:6443 \
--token b29c63.aqvsg0953edz2ozw \
--discovery-token-ca-cert-hash \
sha256:529c64ad705bf356a2efa3c1bdb8181b852e2bc5d46f5d161dee1105e872bae6
```
## [Optional 2] Delete and Re-join Linux Node
Just like what happened with Windows worker node, Linux worker node may also has issue and makes our cluster installation fails, and we may need to remove the failed Linux node from cluster and rejoin it again.
First, get the name of Linux worker node by running this command in the Linux master node:
```bash=1
kubectl get nodes
```
On Linux master node, drain and delete the Linux worker node:
```bash=+
# For example, Linux worker node name is ubuntu-worker-1
kubectl drain ubuntu-worker-1 --delete-local-data --force --ignore-daemonsets
kubectl delete node ubuntu-worker-1
```
On Linux worker node, reset kubeadm:
```bash=+
# Reset kubeadm
kubeadm reset
```
Redo **Step 2** to install Kubernetes in Linux worker node again.
Get the join token again by executing the following command on the Linux master node:
```bash=+
kubeadm token create --print-join-command
```
Afterwards, on Linux worker node, re-join the node
```bash=+
# Join Linux worker node to cluster
kubeadm join 192.168.13.43:6443 \
--token b29c63.aqvsg0953edz2ozw \
--discovery-token-ca-cert-hash \
sha256:529c64ad705bf356a2efa3c1bdb8181b852e2bc5d46f5d161dee1105e872bae6
```
## [Optional 3] Rebuild the Kubernetes Cluster
In some terribly unlucky times, problems keep appearing in our Kubernetes Cluster and nothing we do works. If the situation allows, we may consider to rebuild the Kubernetes cluster from beginning.
First, execute **[Optional 1]** and **[Optional 2]** to remove all Windows and Linux worker nodes.
In Linux master node, check that all worker nodes have been removed and only the master node remains.
```bash=
kubectl get nodes
```
In Linux master node, reset all Kubernetes setting.
```bash=+
kubeadm reset
```
Redo the installation from **Step 3** until **Step 8**.
## [Optional 4] Add more Users to access Kubernetes
Add new user in Ubuntu.
```bash=
sudo adduser newuser
```
Add the `newuser` into `sudo` and `docker` group.
```bash=+
sudo usermod -aG sudo newuser
sudo usermod -aG docker newuser
```
Login as the newuser.
```bash=+
su newuser
```
Add Kubernetes config file to enable access to `kubectl`.
```bash=+
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
```
If Kubernetes config file doesn't exist in `/etc/kubernetes/admin.conf`, then copy the config file from another user.
```bash=+
mkdir -p $HOME/.kube
sudo cp -i /home/administrator/.kube/config $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
```
If all users don't have the Kubernetes config file, then copy the config file from Linux master node.
```bash=+
# Replace [MASTER_NODE_IP] with the IP address of your Linux master node
# Replace [USERNAME] with the username of your Linux master node
scp [USERNAME]@[MASTER_NODE_IP]:/home/[USERNAME]/.kube/config /home/[USERNAME]/.kube/config
# Example:
scp administrator@192.168.0.146:/home/administrator/.kube/config /home/newuser/.kube/config
```
Set the `KUBECONFIG` variable:
```bash=+
export KUBECONFIG=$HOME/.kube/config
```
## References
Below are the references used in this guide, including how-to-install and various troubleshooting manuals.
Future references will be included later.
* [Install Mirantis Docker Container Runtime on Windows Server](https://docs.mirantis.com/docker-enterprise/v3.1/dockeree-products/mcr/mcr-windows.html)
* [Intro to Windows support in Kubernetes](https://kubernetes.io/docs/setup/production-environment/windows/intro-windows-in-kubernetes/)
* [Installing kubeadm](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/)
* [Installing kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl-binary-via-curl)
* [Adding Windows nodes to Kubernetes cluster](https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/adding-windows-nodes/)
* [Kubernetes Dashboard](https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/)
* [Guide for scheduling Windows containers in Kubernetes](https://kubernetes.io/docs/setup/production-environment/windows/user-guide-windows-containers/)
* [Kubernetes and Windows](https://www.chenshaowen.com/blog/add-windows-node-for-k8s.html)
* [Troubleshooting kubeadm](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm/)
* [CoreDNS fails to run in Kubernetes cluster](https://stackoverflow.com/questions/52645473/coredns-fails-to-run-in-kubernetes-cluster)
* https://www.cnblogs.com/aresxin/p/K8S-issue1.html
* https://www.techrunnr.com/how-to-reset-kubernetes-cluster/
Sign in with Wallet
Connect another wallet