---
# System prepended metadata

title: gVisor in rke2
tags: [work]

---

# gVisor in rke2
* gVisor：模擬出一個輕量級的 user-space kernel，可以讓 container 直接使用 user-space kernel。如果 container 自己的 kernel 被破壞掉了，他並不會影響到其他 container。
* 透過 gVisor 建立出來的 container 各自可以擁有自己的 kernel，就不會共用本機的 kernel，對資安上是一大幫助。
![](https://i.imgur.com/GrYvGPd.png)

## gVisor 架構

* Sentry 負責提供 cpu、memory、網路等資源的 systcall。
* Gofer 負責提供檔案系統 mkdir、rm 等的 syscall。
* 但目前還有 108 system call 還不支援，所以不是所有應用系統都能在 gVisor 架構中執行，但持續減少中。
![image](https://hackmd.io/_uploads/ryL1s53W1x.png)

## 安裝 runsc
* 每一個節點都需要下載 runsc，與設定 containerd
```
##判別硬體規格
$ ARCH=$(uname -m); URL=https://storage.googleapis.com/gvisor/releases/release/latest/${ARCH}

$ wget ${URL}/runsc ${URL}/runsc.sha512 ${URL}/containerd-shim-runsc-v1 ${URL}/containerd-shim-runsc-v1.sha512

檢核 runsc 及 containerd-shim-runsc-v1 這二個執行檔
##檢查我們下載下來的有沒有被別人破壞
$ sha512sum -c runsc.sha512 -c containerd-shim-runsc-v1.sha512
runsc: OK
containerd-shim-runsc-v1: OK

##刪除檢查檔
$ rm -f *.sha512

##給rusc跟shim有執行跟讀的權限，並且把它移到環境變數的目錄區裡面
$ chmod a+rx runsc containerd-shim-runsc-v1; sudo mv runsc containerd-shim-runsc-v1 /usr/local/bin
```
## 修改 containerd 設定檔
* rke2 的 containerd 設定檔是在 `/var/lib/rancher/rke2/agent/etc/containerd/config.toml` 這個位置
```
$ sudo cp /var/lib/rancher/rke2/agent/etc/containerd/config.toml /var/lib/rancher/rke2/agent/etc/containerd/config.toml.tmpl
```

```
$ cat <<EOF | sudo tee -a /var/lib/rancher/rke2/agent/etc/containerd/config.toml.tmpl
[plugins."io.containerd.runtime.v1.linux"]
  shim_debug = true
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc]
  runtime_type = "io.containerd.runsc.v1"
EOF
```
* master 重啟服務
```
$ sudo systemctl restart rke2-server.service
```
* worker 重啟服務
```
$ sudo systemctl restart rke2-agent.service
```
* 刪除設定檔
```
$ sudo rm /var/lib/rancher/rke2/agent/etc/containerd/config.toml.tmpl
```

## 建立 RuntimeClass

```
$ cat <<EOF | kubectl apply -f -
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: gvisor
handler: runsc
EOF
```

```
$ kubectl get runtimeclass
NAME     HANDLER   AGE
gvisor   runsc     99s
```
## 測試 gvisor
* 使用 gvisor 建立 container
```
# 需要宣告 RuntimeClass
$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: nginx-gvisor
spec:
  runtimeClassName: gvisor
  containers:
  - name: nginx
    image: nginx
EOF
```
* 進入 nginx-gvisor container 檢查 kernel 版本
```
$ kubectl exec -it nginx-gvisor -- bash
root@nginx-gvisor:/# uname -r
4.4.0
```
* 與本機的 sles15-sp4 kernel 版本不同
```
$ uname -r
5.14.21-150400.22-default
```
* 正常使用 runc 做出的 container 都會共用本機的 kernel
```
$ kubectl run nginx-host --image=nginx

$ kubectl exec -it nginx-host -- bash
root@nginx-host:/# uname -r
5.14.21-150400.22-default
```
* 使用 crictl 命令檢查
```
$ sudo crictl ps -a | grep nginx-gvisor
3c6169e4e0431       bc649bab30d15       19 minutes ago      Running             nginx                              0                   0e5de57592fc4       nginx-gvisor
```
* 是使用 runsc 這個 runtime 啟動的
```
$ sudo crictl inspect 3c6169e4e0431 | grep -A5 runsc
    "runtimeType": "io.containerd.runsc.v1",
    "runtimeOptions": null,
    "config": {
      "metadata": {
        "name": "nginx"
      },
```


## 使用 plan 方式部署
```
---
apiVersion: v1
kind: Secret
metadata:
  name: gvisor
  namespace: cattle-system
type: Opaque
stringData:
  gvisor: "20240206"
  upgrade.sh: |
    #!/bin/sh
    set -e
    secrets=$(dirname $0)
    ARCH=$(uname -m)
    URL=https://storage.googleapis.com/gvisor/releases/release/$(cat $secrets/gvisor)/${ARCH}
    wget ${URL}/runsc ${URL}/runsc.sha512 ${URL}/containerd-shim-runsc-v1 ${URL}/containerd-shim-runsc-v1.sha512
    sha512sum -c runsc.sha512 -c containerd-shim-runsc-v1.sha512
    chmod a+rx runsc containerd-shim-runsc-v1
    sudo mv runsc containerd-shim-runsc-v1 /usr/local/bin
    cat <<-EOF > /var/lib/rancher/rke2/agent/etc/containerd/config.toml.tmpl
    
    {{ template "base" . }}
    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc]
      runtime_type = "io.containerd.runsc.v1"
    EOF
    if systemctl is-active --quiet rke2-server
    then
      echo "Running on a master"
      systemctl restart rke2-server
    else
      echo "Running on a worker"
      systemctl restart rke2-agent
    fi
---
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: gvisor
  namespace: cattle-system
spec:
  concurrency: 1
  nodeSelector:
    matchExpressions:
      - {key: cattle.io/os, operator: In, values: ["linux"]}
  serviceAccountName: system-agent-upgrader
  secrets:
    - name: gvisor
      path: /host/run/system-upgrade/secrets/gvisor
  version: "22.04"
  upgrade:
    image: ubuntu
    command: ["chroot", "/host"]
    args: ["sh", "/run/system-upgrade/secrets/gvisor/upgrade.sh"]
```

* 參考 https://gist.github.com/dgiebert/1c19ab257b4554662399bc75545195db