# Kubeedge
## Install
* https://blog.miniasp.com/post/2020/08/21/Install-Kubernetes-cluster-in-WSL-2-Docker-on-Windows-using-kind
### Install kubeedge

# Download released ver. from github
* https://github.com/kubeedge/kubeedge/releases

### Install kubeadm

# https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sudo sysctl --system

cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF

# Set SELinux in permissive mode (effectively disabling it)
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

sudo systemctl enable --now kubelet

# prerequisite (assume docker is installed)
yum install mosquitto
# Install 
git clone https://github.com/kubeedge/kubeedge.git $GOPATH/src/github.com/kubeedge/kubeedge
cd $GOPATH/src/github.com/kubeedge/kubeedge
make # or `make -C edge edgecore`

# Test running
mosquitto -d -p 1883

# Cloud node config
$GOPATH/src/github.com/kubeedge/kubeedge/cloudcore --defaultconfig > /etc/kubeedge/config/cloudcore.yaml


### Set up cloud side

keadm init --advertise-address=140.113.207.102

# KubeEdge
> Official tutorials
> 1.Kubeadm(Simplified Chinese):
> https://k8smeetup.github.io/docs/admin/kubeadm/
> 2.Kubeedge
> https://github.com/kubeedge/kubeedge/blob/master/docs/setup/setup.md
>
> 3.Tutorial
> https://ithelp.ithome.com.tw/articles/10216352
>
> 4.[k8s訪問外網](https://www.jianshu.com/p/50b930fa7ca3)
>
>5.[https://discuss.kubernetes.io/t/work-node-join-cluster-by-public-ip/5773/6](https://discuss.kubernetes.io/t/work-node-join-cluster-by-public-ip/5773/6)
>
>6.[External IP problem](https://github.com/kubernetes/kubeadm/issues/1390)

[TOC]

## Process
### Prerequisites
1. Docker
2. kubeadm/kubectl
3. Go (Using Go 1.12.9,Go 1.13 will fail somehow)
4. mosquitto (On edge side)
### Cloud part (gemini-11)
###### Install kubeadm
apt-get update && apt-get install -y apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y kubelet kubeadm kubectl
apt-mark hold kubelet kubeadm kubectl

###### Install docker
# Install Docker CE
## Set up the repository:
### Update the apt package index
apt-get update

### Install packages to allow apt to use a repository over HTTPS
apt-get install apt-transport-https ca-certificates curl software-properties-common

### Add Docker’s official GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -

### Add docker apt repository.
add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
stable"

## Install docker ce.
apt-get update
# 指定版本, v18.06.2
apt-get install docker-ce=18.06.2~ce~3-0~ubuntu
# 安裝最新版
# apt-get install docker-ce

# Setup daemon.
## Log 的大小, Storage Driver
cat > /etc/docker/daemon.json <<EOF
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
EOF

mkdir -p /etc/systemd/system/docker.service.d

# Restart docker.
systemctl daemon-reload
systemctl restart docker

> Error:
> 1.Docker daemon service cannot be turned on
> Try:update-rc.d docker enable
> Doesn't work
> 2.[ERROR Swap]: running with swap on is not supported. Please disable swap
> Sol: swapoff -a

###### Create k8s cluster with kubeadm #####
1. Create master node

kubeadm init
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
>init後面的部分是為了把config蓋過去，不然會有certificate不正確的問題(過期了)
2. Check Status (Not-Ready)

kubectl get nodes
kubectl describe node XXX
3. Deploy CNI(Flannel)

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
###### Run Kubeedge ######
4. Clone KubeEdge

git clone https://github.com/kubeedge/kubeedge.git $GOPATH/src/github.com/kubeedge/kubeedge
cd $GOPATH/src/github.com/kubeedge/kubeedge
5. Generate Certificates

$GOPATH/src/github.com/kubeedge/kubeedge/build/tools/certgen.sh genCertAndKey edge
>Error:
Can't load /root/.rnd into RNG
140320946213312:error:2406F079:random number generator:RAND_load_file:Cannot ope n file:../crypto/rand/randfile.c:88:Filename=/root/.rnd
>Solution:
>vi /etc/ssl/openssl.cnf
>comment out RANDFILE = $ENV::HOME/.rnd

Cert pos:/etc/kubeedge/ca
Key pos:/etc/kubeedge/certs
6. Build cloudcore

cd $GOPATH/src/github.com/kubeedge/kubeedge/
make all WHAT=cloudcore

7. Create device model and device CRDs.

cd $GOPATH/src/github.com/kubeedge/kubeedge/build/crds/devices
kubectl create -f devices_v1alpha1_devicemodel.yaml
kubectl create -f devices_v1alpha1_device.yaml
> Error: Unable to connect to the server: dial tcp 192.168.1.105:6443: i/o timeout

8. Copy cloudcore binary and config file

mkdir -p ~/cmd/conf
cd ~/src/github.com/kubeedge/kubeedge/cloud
cp cloudcore ~/cmd/
cp -rf conf/* ~/cmd/conf/
~/cmd/ is current binary startup directory
9. Set cloudcore config file

cd ~/cmd/conf
vim controller.yaml
10. Run cloudcore

sudo ln build/tools/cloudcore.service /etc/systemd/system/cloudcore.service
sudo systemctl daemon-reload
sudo systemctl start cloudcore

### Edge part (Jetson nano)
###### Cloud part ######
> Kubeedge team provides an example of node.json in the project.
1. Copy node.json

mkdir ~/cmd/yaml
cp $GOPATH/src/github.com/kubeedge/kubeedge/build/node.json ~/cmd/yaml
2. Set the node role as "edge"
###### Deploy the edge node #####
3. Create the edge node via kubeadm

kubeadm init
4. Check Status (Not-Ready)

kubectl get nodes
kubectl describe node XXX
>Problem:
>Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")
> Solution:
mv $HOME/.kube $HOME/.kube.bak
mkdir $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
5. Deploy CNI(Flannel)

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

6. Move the certs & ca to Edge
###### Connecting edge node to cloud #####

7. Clone Kubeedge

git clone https://github.com/kubeedge/kubeedge.git $GOPATH/src/github.com/kubeedge/kubeedge
cd $GOPATH/src/github.com/kubeedge/kubeedge

8. Run edge(For ARM structure)

cd $GOPATH/src/github.com/kubeedge/kubeedge/edge
sudo apt-get install gcc-arm-linux-gnueabi
export GOARCH=arm
export GOOS="linux"
export GOARM=8 #Pls give the appropriate arm version of your device
export CGO_ENABLED=1
export CC=arm-linux-gnueabi-gcc
make all WHAT=edge_cross_build
> Error:
> Jetson nano has GOARM version=8
> But GOARM value can only be 5,6,7
> Solution:
> export GOARM=7
>
> Error:
> "make edge_cross_build" failed (no response)
>
> Another try:
> make all WHAT=edgecore
> (But I think this is for AMD structure?)
>
7. Install mosquito

sudo apt-add-repository ppa:mosquitto-dev/mosquitto-ppa
sudo apt-get update

> Error:
> PPA failed
> Solution:
> sudo apt install -y mosquitto
>

9. Move Cloud Keys(ca,certs) to edge

10. Modify network config

mkdir ~/cmd/conf
cp $GOPATH/src/github.com/kubeedge/kubeedge/edge/conf/* ~/cmd/conf
vim ~/cmd/conf/edge.yaml

> Set:
> 1.edgehub.websocket.url -> 140.110.18.39:2219
> 2.edged.podsandbox-image -> kubeedge/pause-arm:3.1
> 3. override-> edge-node(user's name)

11. Run edgecore

cp $GOPATH/src/github.com/kubeedge/kubeedge/edge/edgecore ~/cmd/
cd ~/cmd
./edgecore
# or
nohup ./edgecore > edgecore.log 2>&1 &

> Error:
> ./edgecore:no such file or directory(but the file exists)
>
12. Run edgecore with systemd

sudo ln build/tools/edgecore.service /etc/systemd/system/edgecore.service
sudo systemctl daemon-reload
sudo systemctl start edgecore

13. Check node status

kubectl get nodes

> Error: NotReady
> Guess: NO VPN
> Try: install forticlinet-> no binary for arm64
> Try 2: Use Ubuntu VPN
> sudo apt-get install pptpd
> Error: VPN connection failed
> Check protocal for forticlient

## Problem
1. WSL doesn't support docker
2. Container doesn't support docker
3. Cannot execute edge_core

    ./edge_core

結果
bash: ./edge_core: No such file or directory

No such file or directory 原因:
1. 檔案不存在->已用file指令檢查
2. There is a file by that name, but it's a dangling symbolic link.
應該不是捷徑，已用
3. The file exists, and you can even read it (for example, the command `file shank-linux-120720110-1-bin` displays something like “ELF 32-bit LSB executable …”), and yet when you try to execute it you're told that the file doesn't exist.

原因為第三種，由file指令得知edge_core為32bit arm程式，而jetson nano 為aarch64 (可由uname -a)查詢。

試圖安裝32bit library:
>[https://askubuntu.com/questions/1090351/can-i-run-an-arm32-bit-app-on-an-arm64bit-platform-which-is-running-ubuntu-16-04](https://askubuntu.com/questions/1090351/can-i-run-an-arm32-bit-app-on-an-arm64bit-platform-which-is-running-ubuntu-16-04)
>
試圖安裝ia32-libs:
>[https://askubuntu.com/questions/133389/no-such-file-or-directory-but-the-file-exists](https://askubuntu.com/questions/133389/no-such-file-or-directory-but-the-file-exists)

仍顯示相同結果，無法執行。
可用readelf -d edge_core觀看檔案之詳細描述
file顯示 edge_core的interpretor為 /lib/ld-，與網上絕大部分範例不同

# Kubernetes
大致分成兩塊:Mastere node, Worker node
## Structure

## Master node
分成四大部分: APIServer Scheduler Controller etcd
### APIServer
負責對外提供RESTful的K8s API服務，管理任何與Pod資源有關的指令，因此也跟kubectl直接互動。
### Scheduler
負責把Pod deploy 到合適的node上，input是pods和nodes組成的list，輸出是pod跟node之間的mapping。
### Contorller manager
負責維護pod等等的內部狀態
### etcd
用來儲存各個資源的狀態

## Worker node
分成三大部分:runtime,kube-proxy,kubelet

### runtime
Container的運作環境，如docker
### kube-proxy

### kubelet
Master在每個node上的agent，負責管理維護所有的container。

###### tags: `NCHC`