---
# System prepended metadata

title: Kubeedge
tags: [NCHC]

---

# Kubeedge
## Install
* https://blog.miniasp.com/post/2020/08/21/Install-Kubernetes-cluster-in-WSL-2-Docker-on-Windows-using-kind
### Install kubeedge
```bash=
# Download released ver. from github
* https://github.com/kubeedge/kubeedge/releases
```
### Install kubeadm
```bash=
# https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sudo sysctl --system

cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF

# Set SELinux in permissive mode (effectively disabling it)
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

sudo systemctl enable --now kubelet
```

```bash=
# prerequisite (assume docker is installed)
yum install mosquitto
# Install 
git clone https://github.com/kubeedge/kubeedge.git $GOPATH/src/github.com/kubeedge/kubeedge
cd $GOPATH/src/github.com/kubeedge/kubeedge
make # or `make -C edge edgecore`

# Test running
mosquitto -d -p 1883

# Cloud node config
$GOPATH/src/github.com/kubeedge/kubeedge/cloudcore --defaultconfig > /etc/kubeedge/config/cloudcore.yaml

```
![](https://i.imgur.com/jRQdZcA.png)




### Set up cloud side
```bash=
keadm init --advertise-address=140.113.207.102
```
# KubeEdge
> Official tutorials
> 1.Kubeadm(Simplified Chinese):
> https://k8smeetup.github.io/docs/admin/kubeadm/
> 2.Kubeedge
> https://github.com/kubeedge/kubeedge/blob/master/docs/setup/setup.md
> 
> 3.Tutorial
> https://ithelp.ithome.com.tw/articles/10216352
> 
> 4.[k8s訪問外網](https://www.jianshu.com/p/50b930fa7ca3)
>
>5.[https://discuss.kubernetes.io/t/work-node-join-cluster-by-public-ip/5773/6](https://discuss.kubernetes.io/t/work-node-join-cluster-by-public-ip/5773/6)
>
>6.[External IP problem](https://github.com/kubernetes/kubeadm/issues/1390)


[TOC]

## Process
### Prerequisites
1. Docker
2. kubeadm/kubectl
3. Go (Using Go 1.12.9,Go 1.13 will fail somehow)
4. mosquitto (On edge side)
### Cloud part (gemini-11)
###### Install kubeadm
    apt-get update && apt-get install -y apt-transport-https curl
    curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
    cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
    deb https://apt.kubernetes.io/ kubernetes-xenial main
    EOF
    apt-get update
    apt-get install -y kubelet kubeadm kubectl
    apt-mark hold kubelet kubeadm kubectl
    
###### Install docker
# Install Docker CE
## Set up the repository:
### Update the apt package index
apt-get update

### Install packages to allow apt to use a repository over HTTPS
apt-get install apt-transport-https ca-certificates curl software-properties-common

### Add Docker’s official GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -

### Add docker apt repository.
add-apt-repository \
    "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
    $(lsb_release -cs) \
    stable"

## Install docker ce.
    apt-get update
    # 指定版本, v18.06.2
    apt-get install docker-ce=18.06.2~ce~3-0~ubuntu
    # 安裝最新版
    # apt-get install docker-ce

    # Setup daemon.
    ## Log 的大小, Storage Driver
    cat > /etc/docker/daemon.json <<EOF
    {
      "exec-opts": ["native.cgroupdriver=systemd"],
      "log-driver": "json-file",
      "log-opts": {
        "max-size": "100m"
      },
      "storage-driver": "overlay2"
    }
    EOF

    mkdir -p /etc/systemd/system/docker.service.d

    # Restart docker.
    systemctl daemon-reload
    systemctl restart docker
    
> Error:
> 1.Docker daemon service cannot be turned on
> Try:update-rc.d docker enable
> Doesn't work
> 2.[ERROR Swap]: running with swap on is not supported. Please disable swap
> Sol: swapoff -a

###### Create k8s cluster with kubeadm #####
1. Create master node

       kubeadm init
       mkdir -p $HOME/.kube
       sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
       sudo chown $(id -u):$(id -g) $HOME/.kube/config
>init後面的部分是為了把config蓋過去，不然會有certificate不正確的問題(過期了)
2. Check Status (Not-Ready)

        kubectl get nodes
        kubectl describe node XXX
3. Deploy CNI(Flannel)

       kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
###### Run Kubeedge ######
4. Clone KubeEdge

       git clone https://github.com/kubeedge/kubeedge.git $GOPATH/src/github.com/kubeedge/kubeedge
       cd $GOPATH/src/github.com/kubeedge/kubeedge
5. Generate Certificates
 
         $GOPATH/src/github.com/kubeedge/kubeedge/build/tools/certgen.sh genCertAndKey edge
   >Error:
   Can't load /root/.rnd into RNG
140320946213312:error:2406F079:random number generator:RAND_load_file:Cannot ope                                                n file:../crypto/rand/randfile.c:88:Filename=/root/.rnd
   >Solution:
   >vi /etc/ssl/openssl.cnf
   >comment out RANDFILE = $ENV::HOME/.rnd

   Cert pos:/etc/kubeedge/ca
   Key pos:/etc/kubeedge/certs
6. Build cloudcore

       cd $GOPATH/src/github.com/kubeedge/kubeedge/
       make all WHAT=cloudcore   
      
7. Create device model and device CRDs.

        cd $GOPATH/src/github.com/kubeedge/kubeedge/build/crds/devices
        kubectl create -f devices_v1alpha1_devicemodel.yaml
        kubectl create -f devices_v1alpha1_device.yaml
> Error: Unable to connect to the server: dial tcp 192.168.1.105:6443: i/o timeout

8. Copy cloudcore binary and config file

        mkdir -p ~/cmd/conf
        cd ~/src/github.com/kubeedge/kubeedge/cloud
        cp cloudcore ~/cmd/
        cp -rf conf/* ~/cmd/conf/
   ~/cmd/ is current binary startup directory
9. Set cloudcore config file

        cd ~/cmd/conf
        vim controller.yaml
10. Run cloudcore

        sudo ln build/tools/cloudcore.service /etc/systemd/system/cloudcore.service
        sudo systemctl daemon-reload
        sudo systemctl start cloudcore 

### Edge part (Jetson nano)
###### Cloud part ######
> Kubeedge team provides an example of node.json in the project.        
1. Copy node.json

        mkdir ~/cmd/yaml
        cp $GOPATH/src/github.com/kubeedge/kubeedge/build/node.json ~/cmd/yaml
2. Set the node role as "edge"
###### Deploy the edge node #####
3. Create the edge node via kubeadm

        kubeadm init
4. Check Status (Not-Ready)

        kubectl get nodes
        kubectl describe node XXX
   >Problem:
   >Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")
   > Solution:
    mv $HOME/.kube $HOME/.kube.bak
        mkdir $HOME/.kube
        sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
        sudo chown $(id -u):$(id -g) $HOME/.kube/config
5. Deploy CNI(Flannel)

       kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml 

6. Move the certs & ca to Edge 
###### Connecting edge node to cloud #####

7. Clone Kubeedge

        git clone https://github.com/kubeedge/kubeedge.git $GOPATH/src/github.com/kubeedge/kubeedge
        cd $GOPATH/src/github.com/kubeedge/kubeedge
        
8. Run edge(For ARM structure) 

        cd $GOPATH/src/github.com/kubeedge/kubeedge/edge
        sudo apt-get install gcc-arm-linux-gnueabi
        export GOARCH=arm
        export GOOS="linux"
        export GOARM=8                             #Pls give the appropriate arm version of your device  
        export CGO_ENABLED=1
        export CC=arm-linux-gnueabi-gcc
        make all WHAT=edge_cross_build
> Error:
> Jetson nano has GOARM version=8
> But GOARM value can only be 5,6,7
> Solution:
> export GOARM=7
> 
> Error:
> "make edge_cross_build" failed (no response)
> 
> Another try:
> make all WHAT=edgecore
> (But I think this is for AMD structure?)
> 
7. Install mosquito

        sudo apt-add-repository ppa:mosquitto-dev/mosquitto-ppa
        sudo apt-get update

> Error:
> PPA failed
> Solution:
> sudo apt install -y mosquitto
> 

9. Move Cloud Keys(ca,certs) to edge  

10. Modify network config

        mkdir ~/cmd/conf
        cp $GOPATH/src/github.com/kubeedge/kubeedge/edge/conf/* ~/cmd/conf
        vim ~/cmd/conf/edge.yaml

> Set:
> 1.edgehub.websocket.url -> 140.110.18.39:2219
> 2.edged.podsandbox-image -> kubeedge/pause-arm:3.1
> 3. override-> edge-node(user's name)

11. Run edgecore

        cp $GOPATH/src/github.com/kubeedge/kubeedge/edge/edgecore ~/cmd/
        cd ~/cmd
        ./edgecore
        # or
        nohup ./edgecore > edgecore.log 2>&1 &

> Error:
> ./edgecore:no such file or directory(but the file exists)
> 
12. Run edgecore with systemd

        sudo ln build/tools/edgecore.service /etc/systemd/system/edgecore.service
        sudo systemctl daemon-reload
        sudo systemctl start edgecore
        
13. Check node status

        kubectl get nodes
        
> Error: NotReady
> Guess: NO VPN
> Try: install forticlinet-> no binary for arm64
> Try 2: Use Ubuntu VPN
> sudo apt-get install pptpd
> Error: VPN connection failed 
> Check protocal for forticlient

## Problem
1. WSL doesn't support docker
2. Container doesn't support docker
3. Cannot execute edge_core

    ``` bash=
    ./edge_core
    ```
    結果
    bash: ./edge_core: No such file or directory

    No such file or directory 原因:
    1. 檔案不存在->已用file指令檢查 
    2. There is a file by that name, but it's a dangling symbolic link.
       應該不是捷徑，已用
    3. The file exists, and you can even read it (for example, the command `file shank-linux-120720110-1-bin` displays something like “ELF 32-bit LSB executable …”), and yet when you try to execute it you're told that the file doesn't exist.

       原因為第三種，由file指令得知edge_core為32bit arm程式，而jetson nano 為aarch64 (可由uname -a)查詢。
       
       試圖安裝32bit library:
       >[https://askubuntu.com/questions/1090351/can-i-run-an-arm32-bit-app-on-an-arm64bit-platform-which-is-running-ubuntu-16-04](https://askubuntu.com/questions/1090351/can-i-run-an-arm32-bit-app-on-an-arm64bit-platform-which-is-running-ubuntu-16-04)
       >
       試圖安裝ia32-libs:
       >[https://askubuntu.com/questions/133389/no-such-file-or-directory-but-the-file-exists](https://askubuntu.com/questions/133389/no-such-file-or-directory-but-the-file-exists)

       仍顯示相同結果，無法執行。
       可用readelf -d edge_core觀看檔案之詳細描述
       file顯示 edge_core的interpretor為 /lib/ld-，與網上絕大部分範例不同
       

# Kubernetes 
大致分成兩塊:Mastere node, Worker node
## Structure 
![](https://i.imgur.com/9jf5Xtu.png)

## Master node
分成四大部分: APIServer Scheduler Controller etcd
### APIServer
負責對外提供RESTful的K8s API服務，管理任何與Pod資源有關的指令，因此也跟kubectl直接互動。
### Scheduler
負責把Pod deploy 到合適的node上，input是pods和nodes組成的list，輸出是pod跟node之間的mapping。
### Contorller manager
負責維護pod等等的內部狀態
### etcd
用來儲存各個資源的狀態

## Worker node
分成三大部分:runtime,kube-proxy,kubelet

### runtime
Container的運作環境，如docker
### kube-proxy

### kubelet
Master在每個node上的agent，負責管理維護所有的container。

###### tags: `NCHC`


