# rke2 upgrade 1.30 cilium error

* rke2 升級到 1.30 後 cilium 起不來


* 錯誤訊息

$ kubectl -n kube-system logs cilium-2bgp7 -c mount-cgroup
nsenter: cannot open /hostproc/1/ns/cgroup: Permission denied

## 修復
1. 修改 cilium ds 新增 annotations
2. 在 `cilium-agent`、`mount-cgroup`、`apply-sysctl-overwrites`、`mount-bpf-fs` container 都要新增 `appArmorProfile` 參數

$ kubectl -n kube-system edit ds cilium

   annotations:
      container.apparmor.security.beta.kubernetes.io/cilium-agent: "unconfined"
      container.apparmor.security.beta.kubernetes.io/clean-cilium-state: "unconfined"
      container.apparmor.security.beta.kubernetes.io/mount-cgroup: "unconfined"
      container.apparmor.security.beta.kubernetes.io/apply-sysctl-overwrites: "unconfined"
......
        securityContext:
          appArmorProfile:
            type: Unconfined



* 更改好後 cilium pod 都恢復

$ kubectl -n kube-system get po -l app.kubernetes.io/name=cilium-agent
NAME           READY   STATUS    RESTARTS   AGE
cilium-6ssx4   1/1     Running   0          56s
cilium-fzdfc   1/1     Running   0          56s
cilium-lr8wx   1/1     Running   0          56s
cilium-pbtzx   1/1     Running   0          55s
cilium-v2tn5   1/1     Running   0          56s

## HelmChartConfig 套用直接修復

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-cilium
  namespace: kube-system
spec:
  valuesContent: |-
    podAnnotations:
      container.apparmor.security.beta.kubernetes.io/cilium-agent: "unconfined"
      container.apparmor.security.beta.kubernetes.io/clean-cilium-state: "unconfined"
      container.apparmor.security.beta.kubernetes.io/mount-cgroup: "unconfined"
      container.apparmor.security.beta.kubernetes.io/apply-sysctl-overwrites: "unconfined"

## 參考
https://github.com/rancher/rancher/issues/46726

https://github.com/rancher/rke2-charts/blob/699fe0e2b3006587b98e9b2add8e77fecc2bee7a/charts/rke2-cilium/rke2-cilium/1.16.000/templates/cilium-agent/daemonset.yaml#L95