# gVisor in rke2
* gVisor：模擬出一個輕量級的 user-space kernel，可以讓 container 直接使用 user-space kernel。如果 container 自己的 kernel 被破壞掉了，他並不會影響到其他 container。
* 透過 gVisor 建立出來的 container 各自可以擁有自己的 kernel，就不會共用本機的 kernel，對資安上是一大幫助。

## gVisor 架構

* Sentry 負責提供 cpu、memory、網路等資源的 systcall。
* Gofer 負責提供檔案系統 mkdir、rm 等的 syscall。
* 但目前還有 108 system call 還不支援，所以不是所有應用系統都能在 gVisor 架構中執行，但持續減少中。

## 安裝 runsc
* 每一個節點都需要下載 runsc，與設定 containerd

##判別硬體規格
$ ARCH=$(uname -m); URL=https://storage.googleapis.com/gvisor/releases/release/latest/${ARCH}

$ wget ${URL}/runsc ${URL}/runsc.sha512 ${URL}/containerd-shim-runsc-v1 ${URL}/containerd-shim-runsc-v1.sha512

檢核 runsc 及 containerd-shim-runsc-v1 這二個執行檔
##檢查我們下載下來的有沒有被別人破壞
$ sha512sum -c runsc.sha512 -c containerd-shim-runsc-v1.sha512
runsc: OK
containerd-shim-runsc-v1: OK

##刪除檢查檔
$ rm -f *.sha512

##給rusc跟shim有執行跟讀的權限，並且把它移到環境變數的目錄區裡面
$ chmod a+rx runsc containerd-shim-runsc-v1; sudo mv runsc containerd-shim-runsc-v1 /usr/local/bin

## 修改 containerd 設定檔
* rke2 的 containerd 設定檔是在 `/var/lib/rancher/rke2/agent/etc/containerd/config.toml` 這個位置

$ sudo cp /var/lib/rancher/rke2/agent/etc/containerd/config.toml /var/lib/rancher/rke2/agent/etc/containerd/config.toml.tmpl

$ cat <<EOF | sudo tee -a /var/lib/rancher/rke2/agent/etc/containerd/config.toml.tmpl
[plugins."io.containerd.runtime.v1.linux"]
  shim_debug = true
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc]
  runtime_type = "io.containerd.runsc.v1"
EOF

* master 重啟服務

$ sudo systemctl restart rke2-server.service

* worker 重啟服務

$ sudo systemctl restart rke2-agent.service

* 刪除設定檔

$ sudo rm /var/lib/rancher/rke2/agent/etc/containerd/config.toml.tmpl

## 建立 RuntimeClass

$ cat <<EOF | kubectl apply -f -
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: gvisor
handler: runsc
EOF

$ kubectl get runtimeclass
NAME     HANDLER   AGE
gvisor   runsc     99s

## 測試 gvisor
* 使用 gvisor 建立 container

# 需要宣告 RuntimeClass
$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: nginx-gvisor
spec:
  runtimeClassName: gvisor
  containers:
  - name: nginx
    image: nginx
EOF

* 進入 nginx-gvisor container 檢查 kernel 版本

$ kubectl exec -it nginx-gvisor -- bash
root@nginx-gvisor:/# uname -r
4.4.0

* 與本機的 sles15-sp4 kernel 版本不同

$ uname -r
5.14.21-150400.22-default

* 正常使用 runc 做出的 container 都會共用本機的 kernel

$ kubectl run nginx-host --image=nginx

$ kubectl exec -it nginx-host -- bash
root@nginx-host:/# uname -r
5.14.21-150400.22-default

* 使用 crictl 命令檢查

$ sudo crictl ps -a | grep nginx-gvisor
3c6169e4e0431       bc649bab30d15       19 minutes ago      Running             nginx                              0                   0e5de57592fc4       nginx-gvisor

* 是使用 runsc 這個 runtime 啟動的

$ sudo crictl inspect 3c6169e4e0431 | grep -A5 runsc
    "runtimeType": "io.containerd.runsc.v1",
    "runtimeOptions": null,
    "config": {
      "metadata": {
        "name": "nginx"
      },

## 使用 plan 方式部署

---
apiVersion: v1
kind: Secret
metadata:
  name: gvisor
  namespace: cattle-system
type: Opaque
stringData:
  gvisor: "20240206"
  upgrade.sh: |
    #!/bin/sh
    set -e
    secrets=$(dirname $0)
    ARCH=$(uname -m)
    URL=https://storage.googleapis.com/gvisor/releases/release/$(cat $secrets/gvisor)/${ARCH}
    wget ${URL}/runsc ${URL}/runsc.sha512 ${URL}/containerd-shim-runsc-v1 ${URL}/containerd-shim-runsc-v1.sha512
    sha512sum -c runsc.sha512 -c containerd-shim-runsc-v1.sha512
    chmod a+rx runsc containerd-shim-runsc-v1
    sudo mv runsc containerd-shim-runsc-v1 /usr/local/bin
    cat <<-EOF > /var/lib/rancher/rke2/agent/etc/containerd/config.toml.tmpl
    
    {{ template "base" . }}
    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc]
      runtime_type = "io.containerd.runsc.v1"
    EOF
    if systemctl is-active --quiet rke2-server
    then
      echo "Running on a master"
      systemctl restart rke2-server
    else
      echo "Running on a worker"
      systemctl restart rke2-agent
    fi
---
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: gvisor
  namespace: cattle-system
spec:
  concurrency: 1
  nodeSelector:
    matchExpressions:
      - {key: cattle.io/os, operator: In, values: ["linux"]}
  serviceAccountName: system-agent-upgrader
  secrets:
    - name: gvisor
      path: /host/run/system-upgrade/secrets/gvisor
  version: "22.04"
  upgrade:
    image: ubuntu
    command: ["chroot", "/host"]
    args: ["sh", "/run/system-upgrade/secrets/gvisor/upgrade.sh"]

* 參考 https://gist.github.com/dgiebert/1c19ab257b4554662399bc75545195db