[plugin][CNI] Cilium
===
###### tags: `K8s / plugin`
###### tags: `Kubernetes`, `k8s`, `plugin`, `CNI`, `Cilium`
<br>
[TOC]
<br>
## 背景知識
### [[hackmd][plugin] CNI plugin 概念](https://hackmd.io/8Mg-ug4RSYO4ybK4ate0sA)
常見的 CNI Plugin 有：
| 名稱 | 特點 |
|------------|-----|
| **Flannel** | 最簡單的網路，適合入門 |
| **Calico** | 支援網路策略與 BGP，性能好 |
| **Cilium** | 使用 eBPF 技術，支援高安全與可觀察性 |
| **Weave** | 適合小型叢集 |
| **Canal** | Flannel + Calico 混合方案 |
<br>
## 最小可用配置範例（單節點或簡易叢集）
### 🛠️ Helm 安裝 Cilium 常見參數對照表
> by gpt-4o (2025/06/06)
以下是 **使用 Helm 安裝 Cilium v1.17.x** 時常見的 `--set` 參數說明表，包含每個參數的用途、是否必要，以及推薦的情境：
| 參數名稱 | 範例值 | 必要性 | 說明 |
|----------|--------|--------|------|
| `kubeProxyReplacement` | `true` 或 `"strict"` | ✅ 必要 | 啟用 kube-proxy 替代功能。設定為 `true` 表示啟用 BPF-based service routing，建議搭配 `strict` 使用（完全取代 kube-proxy）。 |
| `k8sServiceHost` | 節點 IP（如 `10.78.26.241`） | ✅ 必要 | 指定 Kubernetes API Server 的 IP（`kube-apiserver`）。Helm 安裝時需指定以取得集群資訊。 |
| `k8sServicePort` | `6443` | ✅ 必要 | Kubernetes API Server 的通訊埠，通常是 6443。 |
| `ipam.mode` | `kubernetes` | ✅ 強烈建議 | 使用 Kubernetes 的 PodCIDR IP 分配機制。適合 kubeadm 初始化的叢集，最穩定簡單。 |
| `operator.replicas` | `1` | ✅ 必要 | Cilium operator pod 的副本數。單節點或小型叢集設為 1 即可。 |
| `securityContext.privileged` | `true` | ✅ 必要 | 允許 Cilium agent 有操作核心 BPF 的特權，否則無法正常運作。 |
| `hubble.relay.enabled` | `true` | ⚙️ 選用 | 啟用 Hubble Relay，讓 Hubble UI 能跨節點收集資料。 |
| `hubble.ui.enabled` | `true` | ⚙️ 選用 | 啟用 Hubble 網頁 UI（預設埠為 NodePort）。適合開發或觀察網路流量時使用。 |
| `cluster.name` | `default` | ⚙️ 建議 | 指定叢集名稱，對於多叢集管理或 Hubble 可觀察性有幫助。 |
| `cluster.id` | `0` | ⚙️ 建議 | 叢集的 ID，用於多叢集或 Cilium ClusterMesh。單叢集可設為 0。 |
---
### ✅ 最小可用配置範例（單節點或簡易叢集）
使用下列指令安裝最小設定的 Cilium：
```
helm install cilium cilium/cilium --version 1.17.2 \
--namespace kube-system \
--set kubeProxyReplacement=true \
--set k8sServiceHost=10.78.26.241 \
--set k8sServicePort=6443 \
--set operator.replicas=1 \
--set ipam.mode=kubernetes \
--set securityContext.privileged=true
```
- #### 若需觀察流量、啟用 UI
可額外加上以下設定以啟用 Hubble：
```
--set hubble.relay.enabled=true \
--set hubble.ui.enabled=true \
--set cluster.name=default \
--set cluster.id=0
```
<br>
## 參數
### kube-proxy-replacement
- [Deprecated Options](https://docs.cilium.io/en/v1.14/operations/upgrade/#deprecated-options)
> The kube-proxy-replacement option’s values strict, partial and disabled are deprecated and will be removed in v1.15. They are replaced by true and false. true corresponds to strict, i.e. enables all kube-proxy replacement features. false disables kube-proxy replacement but allows users to selectively enable each kube-proxy replacement feature individually.
- ### gpt-4o (2025/06/06)
> kubeProxyReplacement 參數變更說明
### 🛑 `kubeProxyReplacement` 的舊值即將淘汰
| 舊值 | 狀態 | 取代方式 |
|------------|---------------|----------|
| `strict` | ✅ **Deprecated**，v1.15 將移除 | ➡️ 改用 `true` |
| `partial` | ✅ Deprecated | ➡️ 改用 `false` 並手動設定 |
| `disabled` | ✅ Deprecated | ➡️ 改用 `false` |
---
### ✅ 建議用法（新寫法）
| 值 | 意義說明 |
|------------|----------|
| `true` | 啟用**所有 kube-proxy 替代功能**，等同於舊的 `strict` |
| `false` | 完全不取代 kube-proxy，但可手動選擇開啟個別功能（例如只開 eBPF service 或 hostport） |
<br>
## 診斷 Cilium 狀態
```
$ cilium status
# 找到一個 Cilium Pod 的名稱
CILIUM_POD=$(kubectl -n kube-system get pods -l k8s-app=cilium -o jsonpath='{.items[0].metadata.name}')
# 執行 status 指令
kubectl -n kube-system exec -it $CILIUM_POD -- cilium status --verbose
```
- 執行 status 指令，同底下：
```
kubectl -n kube-system exec ds/cilium -- cilium-dbg status --verbose
```
### 1️⃣ Cilium 狀態
```bash
$ kubectl exec -n kube-system -it $(kubectl get pod -n kube-system -l k8s-app=cilium -o jsonpath='{.items[0].metadata.name}') -- cilium status
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init), install-cni-binaries (init)
KVStore: Disabled
Kubernetes: Ok 1.33 (v1.33.1) [linux/amd64]
Kubernetes APIs: ["EndpointSliceOrEndpoint", "cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumCIDRGroup", "core/v1::Namespace", "core/v1::Pods", "core/v1::Service", "networking.k8s.io/v1::NetworkPolicy"]
KubeProxyReplacement: True [ens10f0.206 10.78.26.241 fe80::1a31:bfff:fece:1d73 (Direct Routing)]
Host firewall: Disabled
SRv6: Disabled
CNI Chaining: none
CNI Config file: successfully wrote CNI configuration file to /host/etc/cni/net.d/05-cilium.conflist
Cilium: Ok 1.17.4 (v1.17.4-55aecc0f)
NodeMonitor: Listening for events on 88 CPUs with 64x4096 of shared memory
Cilium health daemon: Ok
IPAM: IPv4: 9/254 allocated from 10.244.0.0/24,
IPv4 BIG TCP: Disabled
IPv6 BIG TCP: Disabled
BandwidthManager: Disabled
Routing: Network: Native Host: Legacy
Attach Mode: Legacy TC
Device Mode: veth
Masquerading: BPF [ens10f0.206] 10.244.0.0/16 [IPv4: Enabled, IPv6: Disabled]
Controller Status: 64/64 healthy
Proxy Status: OK, ip 10.244.0.180, 0 redirects active on ports 10000-20000, Envoy: external
Global Identity Range: min 256, max 65535
Hubble: Ok Current/Max Flows: 4095/4095 (100.00%), Flows/s: 23.71 Metrics: Disabled
Encryption: Disabled
Cluster health: 1/1 reachable (2025-06-06T04:00:09Z)
Name IP Node Endpoints
Modules Health: Stopped(0) Degraded(0) OK(70)
```
### 2️⃣ Cilium Service 處理狀態
> 用法: `kubectl -n kube-system exec ds/cilium -- cilium service list`
```bash
$ kubectl exec -n kube-system -it $(kubectl get pod -n kube-system -l k8s-app=cilium -o jsonpath='{.items[0].metadata.name}') -- cilium service list
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init), install-cni-binaries (init)
ID Frontend Service Type Backend
1 10.96.0.1:443/TCP ClusterIP 1 => 10.78.26.241:6443/TCP (active)
2 10.98.183.223:8080/TCP ClusterIP
3 10.99.92.141:443/TCP ClusterIP
5 10.96.0.10:53/UDP ClusterIP
6 10.96.0.10:53/TCP ClusterIP
7 10.96.0.10:9153/TCP ClusterIP
9 10.107.174.231:8080/TCP ClusterIP 1 => 10.244.0.47:8080/TCP (active)
10 0.0.0.0:30977/TCP NodePort 1 => 10.244.0.47:8080/TCP (active)
11 10.78.26.241:30977/TCP NodePort 1 => 10.244.0.47:8080/TCP (active)
12 10.78.26.241:4000/TCP HostPort 1 => 10.244.0.47:8080/TCP (active)
13 0.0.0.0:4000/TCP HostPort 1 => 10.244.0.47:8080/TCP (active)
14 10.97.236.113:443/TCP ClusterIP 1 => 10.78.26.241:4244/TCP (active)
```
- 顯示 API Server 的 ClusterIP (`10.96.0.1:443`) 已經正確映射到實際 API Server (`10.78.26.241:6443`)。
### 3️⃣ 查看有無 ClusterIP 路由
```bash
$ ip rule
9: from all fwmark 0x200/0xf00 lookup 2004
100: from all lookup local
32766: from all lookup main
32767: from all lookup default
$ ip route
default via 10.78.26.1 dev ens10f0.206 onlink
10.78.26.0/24 dev ens10f0.206 proto kernel scope link src 10.78.26.241
10.244.0.0/24 via 10.244.0.180 dev cilium_host proto kernel src 10.244.0.180
10.244.0.180 dev cilium_host proto kernel scope link
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
172.18.0.0/16 dev br-7129cd7c4eb9 proto kernel scope link src 172.18.0.1
172.19.0.0/16 dev br-6808f79b3d88 proto kernel scope link src 172.19.0.1
172.21.0.0/16 dev br-503f39658ea3 proto kernel scope link src 172.21.0.1 linkdown
172.23.0.0/16 dev br-78819992758a proto kernel scope link src 172.23.0.1 linkdown
172.24.0.0/16 dev br-a7dbe5108a1f proto kernel scope link src 172.24.0.1 linkdown
192.168.49.0/24 dev br-a0398ea2e748 proto kernel scope link src 192.168.49.1 linkdown
192.168.58.0/24 dev br-8a6fa0fe395b proto kernel scope link src 192.168.58.1 linkdown
192.168.160.0/20 dev br-c3bad0ab66b5 proto kernel scope link src 192.168.160.1 linkdown
```
<br>
## 卸載 Cilium
```bash=
helm uninstall cilium -n kube-system
# 刪除 CRDs（⚠️ 會清掉 Cilium 所有資料）
kubectl delete crds -l app.kubernetes.io/part-of=cilium
# 清理 CNI 設定檔：
sudo rm -f /etc/cni/net.d/05-cilium.conflist
# 清理 Cilium 執行時的 BPF 檔案系統
sudo umount /var/run/cilium/cgroupv2
sudo rm -rf /var/run/cilium
# 檢查是否還有 cilium interface：
ip link show | grep cilium
# 有的話強制刪掉：
sudo ip link delete cilium_vxlan
sudo ip link delete cilium_host
sudo ip link delete cilium_net
```
<br>
## DNS debug
- 開啟 Cilium 的 DNS visibility 功能進行 debug
```
cilium monitor --type dns
```
- 較新版本的 Cilium CLI (?)
```
cilium hubble observe --type dns
```
```
cilium hubble observe --protocol dns
```
<br>
## 討論
### 移除 Flannel 後，如何安裝 Cilium CNI？
- ### gpt-4o (2025/06/05)
以下是在 Kubernetes 中安裝 Cilium CNI（以取代 Flannel）**的完整步驟，適用於乾淨或已移除 Flannel 的環境：
### 🛠️ 安裝 Cilium CNI 的完整流程
- ### ✅ 前置條件檢查
- **移除 Flannel 完畢（你已經完成）**
- **Kernel 支援 BPF（Linux Kernel 4.9+ 最佳）**
- **每個 Node 上安裝好 `iptables`, `iproute2`, `bpftool`（Cilium 安裝腳本會自動偵測）**
- ### 🧾 步驟一：下載並安裝 Cilium CLI（非必須但建議）
```bash=
# Linux 安裝 Cilium CLI（v0.15+）
export CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-amd64.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-amd64.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-amd64.tar.gz /usr/local/bin
rm cilium-linux-amd64.tar.gz{,.sha256sum}
```
- ### 📦 步驟二：安裝 Cilium 到叢集
你可以選擇以下兩種方式之一：
- #### ✅ 方法 A：使用 cilium install（推薦，簡單、會自動偵測）
```bash
cilium install
```
你可以加入參數：
```bash
cilium install --version 1.15.4
```
- #### 🧾 方法 B：用 kubectl 安裝 YAML（離線環境或手動部署）
```bash
kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.15.4/install/kubernetes/quick-install.yaml
```
可改用你要的版本，如 v1.15.4、v1.14.7 等。
- ### ✅ 步驟三：驗證安裝狀態
```bash
cilium status
```
或直接用 kubectl：
```bash
kubectl -n kube-system get pods -l k8s-app=cilium
kubectl get daemonset cilium -n kube-system
```
- ### 🔧 步驟四：啟用 Hubble（可視化 Cilium 流量）
```bash
cilium hubble enable
cilium hubble port-forward &
cilium hubble status
```
然後瀏覽器開啟： http://localhost:12000
- ### 🧪 步驟五：測試網路連通性
```bash
kubectl create deployment nginx --image=nginx
kubectl expose deployment nginx --port 80
kubectl run curl --image=curlimages/curl -it --rm -- curl http://nginx
```
- ### 🧹 建議：Flannel 移除後 Pod IP 可能變動
若之前有部署 StatefulSet / PV / PVC 的應用，建議重建：
```bash
kubectl delete pod --all -A
```
讓它們以 Cilium 網路重新配置。
---
如果你有安裝 helm、要客製化安裝（啟用 kube-proxy-free, IPv6, eBPF NAT 等），也可以用：
```bash
helm repo add cilium https://helm.cilium.io/
helm install cilium cilium/cilium --version 1.15.4 \
--namespace kube-system \
--set kubeProxyReplacement=true \
--set k8sServiceHost=<API_SERVER_IP> \
--set k8sServicePort=6443
```
需要我幫你根據叢集狀況（例如 kube-proxy 有無啟用）來推薦最佳設定嗎？
<br>
### 安裝完 Cilium 後的狀態
- ### Question
```
$ cilium status
/¯¯\
/¯¯\__/¯¯\ Cilium: OK
\__/¯¯\__/ Operator: OK
/¯¯\__/¯¯\ Envoy DaemonSet: OK
\__/¯¯\__/ Hubble Relay: disabled
\__/ ClusterMesh: disabled
DaemonSet cilium Desired: 1, Ready: 1/1, Available: 1/1
DaemonSet cilium-envoy Desired: 1, Ready: 1/1, Available: 1/1
Deployment cilium-operator Desired: 1, Ready: 1/1, Available: 1/1
Containers: cilium Running: 1
cilium-envoy Running: 1
cilium-operator Running: 1
clustermesh-apiserver
hubble-relay
Cluster Pods: 2/15 managed by Cilium
Helm chart version: 1.17.2
Image versions cilium quay.io/cilium/cilium:v1.17.2@sha256:3c4c9932b5d8368619cb922a497ff2ebc8def5f41c18e410bcc84025fcd385b1: 1
cilium-envoy quay.io/cilium/cilium-envoy:v1.31.5-1741765102-efed3defcc70ab5b263a0fc44c93d316b846a211@sha256:377c78c13d2731f3720f931721ee309159e782d882251709cb0fac3b42c03f4b: 1
cilium-operator quay.io/cilium/operator-generic:v1.17.2@sha256:81f2d7198366e8dec2903a3a8361e4c68d47d19c68a0d42f0b7b6e3f0523f249: 1
```
- ### gpt-4o (2025/06/05)
### Cilium 已成功啟用後的狀態說明
🎉 現在看起來 **Cilium 已經成功安裝並正常運作了！**
從你的 `cilium status` 結果來看：
---
### ✅ 關鍵狀態指標都已就緒：
| 組件 | 狀態 |
|------------------|----------|
| Cilium DaemonSet | OK（1/1）|
| Cilium Operator | OK（1/1）|
| Envoy DaemonSet | OK（1/1）|
- **Containers 全部 Running**
- 沒有 Pending/Error/Warning
- 版本為 Cilium **v1.17.2**
---
### ⚠️ 小提醒
```
Cluster Pods: 2/15 managed by Cilium
```
表示目前叢集中的 Pod 還沒有全部被 Cilium 接管（可能是因為：
- 某些 Pod 尚未重啟取得新的 CNI
- 或有些 Pod 是系統組件未使用 CNI）
建議你可以執行以下命令，確認現有 Pod 是否都成功套用了 Cilium：
```
kubectl get pods -A -o wide | grep -v cilium
```
觀察是否：
- IP 欄位為空（表示沒拿到 CNI）
- Pod Card stuck in `Pending` 或 `ContainerCreating`
---
### 🛠️ 若要讓全部 Pod 都使用 Cilium 網路：
執行強制重啟所有 Pod：
```
kubectl delete pod --all -A
```
> 請確保你的應用具備適當的 `Deployment`, `StatefulSet`, `ReplicaSet`，否則資料會遺失。
---
### 🚀 下一步你可以考慮：
| 功能 | 命令 |
|---------------|------|
| 啟用 Hubble（流量觀察） | `cilium hubble enable && cilium hubble port-forward &` |
| 啟用 kube-proxy-free 模式 | `cilium install --set kubeProxyReplacement=true`（需重新安裝）|
| 查看流量圖 | 瀏覽器開啟 http://localhost:12000 |
---
如果你打算接下來：
- **驗證 Pod 之間通訊**
- **開啟 ClusterIP / LoadBalancer / NodePort**
- **使用 NetworkPolicy**
也可以告訴我，我可以一步步協助你測試與設定。
<br>
### 安裝完 Cilium 後，kubelet 需要重啟？
- ### gpt-4o (2025/06/05)
> 是否需要重啟 kubelet？
一般情況下，**你不需要手動重啟 `kubelet`**，但有幾個特定情境下 **建議或必須重啟**，以下為判斷標準：
---
### 🔍 **建議重啟 kubelet 的情況**
| 情境 | 是否需要重啟 | 原因 |
|------|---------------|------|
| 你剛剛 **從 Flannel 換成 Cilium**，並已刪除原 `/etc/cni/net.d/*` 與 `/run/flannel/*` 等設定 | ✅ 建議重啟 | 確保 `kubelet` 重新載入最新 CNI 設定，否則可能繼續嘗試使用舊 CNI 導致 Pod 卡在 `ContainerCreating` |
| 新裝 CNI Plugin 但 Pod 拿不到 IP | ✅ 必須重啟 | CNI 配置沒被套用，通常 `journalctl -u kubelet` 可見錯誤訊息 |
| 所有 Pod 卡在 `Pending` 或 `ContainerCreating`，無法建立網路 | ✅ 建議重啟 | 可能是 `kubelet` 未偵測到新的網路插件 |
| `ip link` 仍看到舊的 `cni0`, `flannel.1`，即使已移除 | ✅ 建議重啟 | 這些是舊 CNI 建立的橋接裝置，重新啟動會清除殘留 |
| 只是安裝完 Cilium，Pod 網路正常（如你目前狀態） | ❌ 不需重啟 | 表示 `kubelet` 成功與新 CNI（Cilium）整合 |
---
### 🛠️ 如何安全地重啟 kubelet
在每個 Node 上執行：
```
sudo systemctl restart kubelet
```
可搭配：
```
watch -n1 "kubectl get pods -A -o wide"
```
觀察 pod 是否短暫重啟、是否 IP 正常配置。
---
### ✅ 小結：
你目前的狀態（Cilium: OK，Pod 有 IP）**不需要立即重啟 kubelet**。
但如果你懷疑還有一些殘留的 Flannel 設定或 `cni0` 網橋沒清乾淨，**重啟 kubelet 是安全又建議的動作**。
需要我幫你寫一段 script，自動檢查是否需要重啟 kubelet 嗎？
<br>
### Pod 在 Cilium 安裝後仍 DNS 解析失敗的診斷與解法
- ### Question
```
$ kubectl exec -it pod/ubuntu-2204-net-tools-pod -- bash
root@ubuntu-2204-net-tools-pod:/# apt update
Ign:1 http://security.ubuntu.com/ubuntu jammy-security InRelease
Ign:2 http://archive.ubuntu.com/ubuntu jammy InRelease
Ign:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
Ign:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
Ign:2 http://archive.ubuntu.com/ubuntu jammy InRelease
Ign:1 http://security.ubuntu.com/ubuntu jammy-security InRelease
Ign:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
Ign:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
Ign:2 http://archive.ubuntu.com/ubuntu jammy InRelease
Ign:1 http://security.ubuntu.com/ubuntu jammy-security InRelease
Ign:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
Ign:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
Err:2 http://archive.ubuntu.com/ubuntu jammy InRelease
Temporary failure resolving 'archive.ubuntu.com'
Err:1 http://security.ubuntu.com/ubuntu jammy-security InRelease
Temporary failure resolving 'security.ubuntu.com'
Err:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
Temporary failure resolving 'archive.ubuntu.com'
Err:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
Temporary failure resolving 'archive.ubuntu.com'
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
17 packages can be upgraded. Run 'apt list --upgradable' to see them.
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/jammy/InRelease Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/jammy-updates/InRelease Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/jammy-backports/InRelease Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/jammy-security/InRelease Temporary failure resolving 'security.ubuntu.com'
W: Some index files failed to download. They have been ignored, or old ones used instead.
```
- ### gpt-4o (2025/06/05)
`ubuntu-2204-net-tools-pod` 是在 **Cilium 安裝之後建立的**，那麼它的 DNS 問題不是因為「舊的 CNI 錯誤殘留」，我們需要往 **Cilium + DNS 解析路徑** 更深入排查。
---
### ✅ 分析流程：Pod 建立後 DNS 還失敗，可能的原因有三：
#### ❶ `/etc/resolv.conf` 指向錯誤的 DNS IP
請在 pod 中執行：
```
cat /etc/resolv.conf
```
你應該會看到類似：
```
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
```
- 如果 `nameserver` 是不存在的 IP（例如錯誤的 flannel 網段或無對應 kube-dns），就會 DNS failure。
- 正確值應該是 `kube-dns` 的 ClusterIP。
查出正確值：
```
kubectl get svc -n kube-system kube-dns
```
---
#### ❷ Cilium 沒有正確處理 DNS 封包
DNS 是 UDP/53，Cilium 有時會對其做特殊處理（特別是在 kube-proxy replacement 模式下）。
請確認是否已安裝 `coredns` 且 pod 正常：
```
kubectl get pods -n kube-system -l k8s-app=kube-dns
```
並且它 **已 Ready 且 Running**。
你也可以進入 busybox 之類的 pod 測試：
```
kubectl run dns-check --rm -it --image=busybox --restart=Never -- nslookup archive.ubuntu.com
```
---
#### ❸ Node DNS 轉發設定錯誤（例如 `resolv.conf` 錯誤或缺少 `systemd-resolved`）
在該 node 上檢查：
```
cat /etc/resolv.conf
```
確認 Node 本身是否能解析：
```
nslookup archive.ubuntu.com
```
---
### 🔧 你可以嘗試的補救操作
#### ✅ 方法一：顯式重建一個 Pod 測試 DNS（避開原 YAML）
```
kubectl run dns-check --image=ubuntu:22.04 -it --rm -- bash
# 然後在容器裡面執行
apt update
```
如果這個新的 pod 也不能連線，那問題是全域的（Cilium 還沒正確處理 DNS 封包）。
---
### 🛠 若仍失敗，可考慮的 Cilium DNS 強化設定：
你也可以用 CLI 重裝 Cilium 並開啟 `enableRemoteNodeIdentity=false` 或其他 DNS 修復選項，例如：
```
cilium uninstall
cilium install \
--set kubeProxyReplacement=disabled \
--set enableRemoteNodeIdentity=false \
--set hubble.enabled=true
```
---
### ✅ 建議行動（請幫我貼這幾個結果）
1. Pod 中 `/etc/resolv.conf` 的內容
2. `kubectl get svc -n kube-system kube-dns` 的結果
3. 你跑以下測試的輸出（我們可以馬上判斷問題範圍）：
```
kubectl run dns-check --rm -it --image=busybox --restart=Never -- nslookup archive.ubuntu.com
```
我可以根據這些資訊，立即告訴你是哪一層出錯。要我協助逐步排查嗎？
<br>
### Cilium 安裝後 Pod DNS 失效問題診斷與解法
- ### Question
```
# cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5
```
```
$ kubectl get svc -n kube-system kube-dns
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 10
```
- ### gpt-4o (2025/06/05)
### ✅ 目前 DNS 基本設定是正確的：
- Pod 中 `/etc/resolv.conf` 內容為：
```
nameserver 10.96.0.10
```
- `kube-dns` 的 service 存在且 ClusterIP 正是 `10.96.0.10`，對應正確。
- 然而，**Pod 裡仍然無法解析 archive.ubuntu.com**
---
## ❗ 現在的推論是：
> **Cilium 尚未正確處理 DNS 流量轉送（UDP/53），導致 pod 雖然 DNS 設定正確，但 DNS 封包無法送達 CoreDNS Pod。**
這在某些環境下很常見，例如：
- Cilium 安裝時啟用了 `kubeProxyReplacement: true`，但系統未正確設 routing rule。
- 或 Cilium 的 BPF 還未初始化正確。
---
## ✅ 請執行以下測試，以確認 DNS 是否能正常連線
```
kubectl run dns-check --rm -it --image=busybox --restart=Never -- nslookup archive.ubuntu.com
```
成功的話會顯示：
```
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
Name: archive.ubuntu.com
Address 1: 91.189.91.39
```
失敗則會出現：
```
nslookup: can't resolve 'archive.ubuntu.com'
```
---
## 🛠️ 若失敗，建議你這樣修復：
### 選項 A：**使用安全設定重裝 Cilium（關閉 kube-proxy replacement）**
```
cilium uninstall
cilium install \
--set kubeProxyReplacement=disabled \
--set enableRemoteNodeIdentity=false \
--set hubble.enabled=true
```
這會關閉一部分進階 BPF 功能，改用原本 kube-proxy 處理部分路由，對 DNS 較穩定。
---
### 選項 B：開啟 Cilium 的 DNS visibility 功能進行 debug
```
cilium monitor --type dns
```
這樣可以即時監看是否有 DNS 封包嘗試送出。
---
如果你跑了 `dns-check` 測試後提供結果，我就能 100% 確定是 Cilium 的哪一層未接好。是否現在幫你確認？
<br>
<hr>
<br>
### 如何查看當前 Cilium 的設定值
- ### gpt-4o (2025/06/05)
你可以透過以下幾種方式來查看當前安裝的 Cilium 設定值（Helm Values）：
---
### ✅ 方法一：使用 Cilium CLI 查看設定
```
cilium status --verbose
```
這會列出一些重要的安裝參數，例如：
- `KubeProxyReplacement: disabled`
- `EnableRemoteNodeIdentity: false`
- `Hubble: enabled`
- IPAM 模式、Node IP、eBPF 狀態、KubeProxy 狀態等
---
### ✅ 方法二：透過 Helm 查詢已安裝的 values
如果你是透過 `helm` 安裝或 `cilium install`（底層其實也會建 Helm release），你可以這樣查：
```
helm get values cilium -n kube-system
```
或顯示所有（包含預設值與覆寫值）：
```
helm get values cilium -n kube-system --all
```
---
### ✅ 方法三：查看 Cilium 的 ConfigMap（有時含設定）
```
kubectl get configmap cilium-config -n kube-system -o yaml
```
這裡會列出 Cilium 運行期的設定，例如：
```yaml
enable-ipv4: "true"
kube-proxy-replacement: "disabled"
enable-remote-node-identity: "false"
```
⚠️ 有些設定（如 hubble.enabled）不會反映在這裡，但大部分核心設定會出現在 ConfigMap 中。
---
### ✅ 方法四（進階）：查看 Helm Release Secrets（含完整 Chart 設定）
```bash
kubectl get secret -n kube-system -l owner=helm,name=cilium
kubectl get secret <cilium-release-secret-name> -n kube-system -o yaml | base64 -d
```
這可以看到 helm install 當下的原始 values.yaml（通常存在於 release 的 Secret 中）
---
### 🧪 範例
```bash
$ helm get values cilium -n kube-system
USER-SUPPLIED VALUES:
enableRemoteNodeIdentity: false
hubble:
enabled: true
kubeProxyReplacement: disabled
```
---
### 實測方法一：`cilium status --verbose`
```
$ cilium status --verbose
/¯¯\
/¯¯\__/¯¯\ Cilium: OK
\__/¯¯\__/ Operator: OK
/¯¯\__/¯¯\ Envoy DaemonSet: OK
\__/¯¯\__/ Hubble Relay: OK
\__/ ClusterMesh: disabled
DaemonSet cilium Desired: 5, Ready: 5/5, Available: 5/5
DaemonSet cilium-envoy Desired: 5, Ready: 5/5, Available: 5/5
Deployment cilium-operator Desired: 2, Ready: 2/2, Available: 2/2
Deployment hubble-relay Desired: 1, Ready: 1/1, Available: 1/1
Deployment hubble-ui Desired: 1, Ready: 1/1, Available: 1/1
Containers: cilium Running: 5
cilium-envoy Running: 5
cilium-operator Running: 2
clustermesh-apiserver
hubble-relay Running: 1
hubble-ui Running: 1
Cluster Pods: 91/94 managed by Cilium
Helm chart version: 1.17.2
Image versions cilium quay.io/cilium/cilium:v1.17.2@sha256:3c4c9932b5d8368619cb922a497ff2ebc8def5f41c18e410bcc84025fcd385b1: 5
cilium-envoy quay.io/cilium/cilium-envoy:v1.31.5-1741765102-efed3defcc70ab5b263a0fc44c93d316b846a211@sha256:377c78c13d2731f3720f931721ee309159e782d882251709cb0fac3b42c03f4b: 5
cilium-operator quay.io/cilium/operator-generic:v1.17.2@sha256:81f2d7198366e8dec2903a3a8361e4c68d47d19c68a0d42f0b7b6e3f0523f249: 2
hubble-relay quay.io/cilium/hubble-relay:v1.17.2@sha256:42a8db5c256c516cacb5b8937c321b2373ad7a6b0a1e5a5120d5028433d586cc: 1
hubble-ui quay.io/cilium/hubble-ui-backend:v0.13.2@sha256:a034b7e98e6ea796ed26df8f4e71f83fc16465a19d166eff67a03b822c0bfa15: 1
hubble-ui quay.io/cilium/hubble-ui:v0.13.2@sha256:9e37c1296b802830834cc87342a9182ccbb71ffebb711971e849221bd9d59392: 1
```
---
### 實測方法二：`helm get values cilium -n kube-system`
```yaml=
$ helm get values cilium -n kube-system
USER-SUPPLIED VALUES:
autoDirectNodeRoutes: true
bpf:
masquerade: true
hubble:
relay:
enabled: true
ui:
enabled: true
frontend:
server:
ipv6:
enabled: false
ipam:
mode: kubernetes
ipv4NativeRoutingCIDR: 10.244.0.0/16
k8sServiceHost: 10.78.26.220
k8sServicePort: 6443
kubeProxyReplacement: true
routingMode: native
```
---
### 實測方法三：`kubectl get configmap cilium-config -n kube-system -o yaml`
```yaml=
$ kubectl get configmap cilium-config -n kube-system -o yaml
apiVersion: v1
data:
agent-not-ready-taint-key: node.cilium.io/agent-not-ready
arping-refresh-period: 30s
auto-direct-node-routes: "true"
bpf-distributed-lru: "false"
bpf-events-drop-enabled: "true"
bpf-events-policy-verdict-enabled: "true"
bpf-events-trace-enabled: "true"
bpf-lb-acceleration: disabled
bpf-lb-algorithm-annotation: "false"
bpf-lb-external-clusterip: "false"
bpf-lb-map-max: "65536"
bpf-lb-mode-annotation: "false"
bpf-lb-sock: "false"
bpf-lb-source-range-all-types: "false"
bpf-map-dynamic-size-ratio: "0.0025"
bpf-policy-map-max: "16384"
bpf-root: /sys/fs/bpf
cgroup-root: /run/cilium/cgroupv2
cilium-endpoint-gc-interval: 5m0s
cluster-id: "0"
cluster-name: default
clustermesh-enable-endpoint-sync: "false"
clustermesh-enable-mcs-api: "false"
cni-exclusive: "true"
cni-log-file: /var/run/cilium/cilium-cni.log
custom-cni-conf: "false"
datapath-mode: veth
debug: "false"
debug-verbose: ""
default-lb-service-ipam: lbipam
direct-routing-skip-unreachable: "false"
dnsproxy-enable-transparent-mode: "true"
dnsproxy-socket-linger-timeout: "10"
egress-gateway-reconciliation-trigger-interval: 1s
enable-auto-protect-node-port-range: "true"
enable-bpf-clock-probe: "false"
enable-bpf-masquerade: "true"
enable-endpoint-health-checking: "true"
enable-endpoint-lockdown-on-policy-overflow: "false"
enable-experimental-lb: "false"
enable-health-check-loadbalancer-ip: "false"
enable-health-check-nodeport: "true"
enable-health-checking: "true"
enable-hubble: "true"
enable-internal-traffic-policy: "true"
enable-ipv4: "true"
enable-ipv4-big-tcp: "false"
enable-ipv4-masquerade: "true"
enable-ipv6: "false"
enable-ipv6-big-tcp: "false"
enable-ipv6-masquerade: "true"
enable-k8s-networkpolicy: "true"
enable-k8s-terminating-endpoint: "true"
enable-l2-neigh-discovery: "true"
enable-l7-proxy: "true"
enable-lb-ipam: "true"
enable-local-redirect-policy: "false"
enable-masquerade-to-route-source: "false"
enable-metrics: "true"
enable-node-selector-labels: "false"
enable-non-default-deny-policies: "true"
enable-policy: default
enable-policy-secrets-sync: "true"
enable-runtime-device-detection: "true"
enable-sctp: "false"
enable-source-ip-verification: "true"
enable-svc-source-range-check: "true"
enable-tcx: "true"
enable-vtep: "false"
enable-well-known-identities: "false"
enable-xt-socket-fallback: "true"
envoy-access-log-buffer-size: "4096"
envoy-base-id: "0"
envoy-keep-cap-netbindservice: "false"
external-envoy-proxy: "true"
health-check-icmp-failure-threshold: "3"
http-retry-count: "3"
hubble-disable-tls: "false"
hubble-export-file-max-backups: "5"
hubble-export-file-max-size-mb: "10"
hubble-listen-address: :4244
hubble-socket-path: /var/run/cilium/hubble.sock
hubble-tls-cert-file: /var/lib/cilium/tls/hubble/server.crt
hubble-tls-client-ca-files: /var/lib/cilium/tls/hubble/client-ca.crt
hubble-tls-key-file: /var/lib/cilium/tls/hubble/server.key
identity-allocation-mode: crd
identity-gc-interval: 15m0s
identity-heartbeat-timeout: 30m0s
install-no-conntrack-iptables-rules: "false"
ipam: kubernetes
ipam-cilium-node-update-rate: 15s
iptables-random-fully: "false"
ipv4-native-routing-cidr: 10.244.0.0/16
k8s-require-ipv4-pod-cidr: "false"
k8s-require-ipv6-pod-cidr: "false"
kube-proxy-replacement: "true"
kube-proxy-replacement-healthz-bind-address: ""
max-connected-clusters: "255"
mesh-auth-enabled: "true"
mesh-auth-gc-interval: 5m0s
mesh-auth-queue-size: "1024"
mesh-auth-rotated-identities-queue-size: "1024"
monitor-aggregation: medium
monitor-aggregation-flags: all
monitor-aggregation-interval: 5s
nat-map-stats-entries: "32"
nat-map-stats-interval: 30s
node-port-bind-protection: "true"
nodeport-addresses: ""
nodes-gc-interval: 5m0s
operator-api-serve-addr: 127.0.0.1:9234
operator-prometheus-serve-addr: :9963
policy-cidr-match-mode: ""
policy-secrets-namespace: cilium-secrets
policy-secrets-only-from-secrets-namespace: "true"
preallocate-bpf-maps: "false"
procfs: /host/proc
proxy-connect-timeout: "2"
proxy-idle-timeout-seconds: "60"
proxy-initial-fetch-timeout: "30"
proxy-max-concurrent-retries: "128"
proxy-max-connection-duration-seconds: "0"
proxy-max-requests-per-connection: "0"
proxy-xff-num-trusted-hops-egress: "0"
proxy-xff-num-trusted-hops-ingress: "0"
remove-cilium-node-taints: "true"
routing-mode: native
service-no-backend-response: reject
set-cilium-is-up-condition: "true"
set-cilium-node-taints: "true"
synchronize-k8s-nodes: "true"
tofqdns-dns-reject-response-code: refused
tofqdns-enable-dns-compression: "true"
tofqdns-endpoint-max-ip-per-hostname: "1000"
tofqdns-idle-connection-grace-period: 0s
tofqdns-max-deferred-connection-deletes: "10000"
tofqdns-proxy-response-max-delay: 100ms
tunnel-protocol: vxlan
tunnel-source-port-range: 0-0
unmanaged-pod-watcher-interval: "15"
vtep-cidr: ""
vtep-endpoint: ""
vtep-mac: ""
vtep-mask: ""
write-cni-conf-when-ready: /host/etc/cni/net.d/05-cilium.conflist
kind: ConfigMap
metadata:
annotations:
meta.helm.sh/release-name: cilium
meta.helm.sh/release-namespace: kube-system
creationTimestamp: "2025-04-07T03:16:00Z"
labels:
app.kubernetes.io/managed-by: Helm
name: cilium-config
namespace: kube-system
resourceVersion: "5379"
uid: eed0fd0f-0959-4b3d-87f9-feb3e5b92406
```
<br>
<hr>
<br>
### Cilium 的 eBPF 程式在 NAT 表運作機制
- ### gpt-4o (2025/06/14)
Cilium 是一個基於 eBPF 的 Kubernetes 網路外掛，能提供高效能與安全性的網路連接、服務發現與網路策略控制。其 eBPF 程式可以取代傳統 `iptables` 的 NAT 處理，特別是在 **Service NAT**（Kubernetes Service ClusterIP / NodePort）與 **Pod NAT**（SNAT/masquerade）方面。
---
### ✅ Cilium 使用 eBPF 實作 NAT 的關鍵位置
#### 一、使用的 eBPF Hook Points
Cilium 並不依賴傳統的 `iptables -t nat`，而是透過以下 eBPF 程式掛載點完成 NAT：
| eBPF Attach Point | 對應 Linux 網路堆疊階段 | 功能角色 |
| ------------------------ | ---------------- | -------------------- |
| `tc`（traffic control） | L2/L3 網路封包進出介面前後 | Pod 出入封包處理、SNAT/DNAT |
| `xdp`（eXpress Data Path） | 封包進入網卡最早階段（可選） | 加速接收處理、過濾非法封包 |
| `cgroup` socket hooks | 套接字層，與應用交互 | 透明代理、原始位址追蹤（透明 NAT） |
---
### 🧠 NAT 實作機制解說（以 Kubernetes Service 為例）
#### 1. 服務封包轉發（Service NAT）
K8s Service（ClusterIP/NodePort）會透過 NAT 將封包從 `Service IP:Port` 轉向實際 `Pod IP:Port`，Cilium 在此過程中：
* 攔截進入 node 的封包（入口：`tc ingress` on external interface）
* 查表（eBPF map）決定後端 Pod（類似 kube-proxy 的 conntrack+iptables 行為）
* 改寫封包目的 IP/Port（DNAT）
* 記錄映射關係（透過 eBPF conntrack map）
📌 **特色**：
* 使用 `eBPF Maps` 快速查詢 Service Endpoints
* 自帶 load balancing hash 策略（Maglev / Random / RR）
---
#### 2. Pod 出站 SNAT（Masquerade）
若 Pod 發送流量到集群外部（如 Internet），Cilium 預設會進行 SNAT：
* `tc egress`（Pod 的 veth interface）處理封包出站
* 檢查目的地是否屬於 cluster CIDR
* 若非 cluster traffic → 改寫 source IP（SNAT）
* 記錄 conntrack 映射關係
📌 eBPF conntrack table 包含：
* `src IP:port`
* `dst IP:port`
* `translated src IP:port`
* NAT direction (`ingress/egress`)
* timeout
---
### 🧪 與 `iptables -t nat` 差異
| 功能 | 傳統 iptables NAT | Cilium eBPF NAT |
| ----- | ------------------------ | ------------------------ |
| 查詢效率 | 逐條規則比對 | 哈希查表（O(1)） |
| 狀態追蹤 | conntrack table (kernel) | eBPF map 中實作 conntrack |
| 可觀察性 | iptables log, conntrack | `cilium monitor`, Hubble |
| 多租戶隔離 | 較困難 | 更細粒度（per endpoint）控制 |
| 彈性 | 固定 chain/table 結構 | 自定義程式邏輯，可配合自定義策略 |
---
### 🔍 如何觀察 Cilium eBPF NAT 行為？
可使用以下工具觀察：
* `cilium bpf nat list`：列出 NAT 映射表
* `cilium bpf ct list`：查看 conntrack 資料
* `cilium monitor`：封包處理 trace（含 NAT 改寫資訊）
* `hubble observe`：服務與 Pod 流量觀察（內建 eBPF trace 資訊）
---
### 🧩 延伸：NAT Bypass 與 Direct Routing 模式
Cilium 支援兩種方式避免 NAT：
* **Direct Routing（DR）模式**：Pod IP 可直接被其他節點存取，無須 NAT
* **External IP policy**：可設定不對某些目的地進行 SNAT（例如指定的 CIDR）
---
如你有特定情境（例如：NodePort、外部 ingress、Pod 跨 node 通訊）想深入看 eBPF 對應程式碼或 maps，我可以補上更具體細節。需要嗎？
<br>
<hr>
<br>
### 如果封包無法連到外部網路，要如何透過 Cilium 指令查詢?
- ### gpt-4o (2025/06/14)
如果 Pod 或節點無法連到外部網路，在使用 **Cilium** 的情境下，你可以透過以下一系列指令與檢查點來診斷是哪一層出了問題（如：封包沒 NAT、路由錯誤、conntrack 錯誤等）。
---
### 🔍 一、檢查封包是否被 SNAT（Cilium NAT/Conntrack）
#### 1. 查看 NAT 表現況
```bash
cilium bpf nat list
```
* 這會列出 eBPF 中紀錄的 SNAT 映射，例如：
```
TCP OUT 10.244.1.10:40000 → 8.8.8.8:53 SNAT 192.168.1.100:30000
```
#### 2. 查看 conntrack 狀態
```bash
cilium bpf ct list
```
* 看連線是否有建立，有無 `REPLY` 或 `ESTABLISHED` 狀態。
* 若連線卡在 `NEW` 或 `SYN_SENT`，代表回應沒進來。
---
### 🛰️ 二、觀察封包流向與 NAT 行為
#### 3. 實時監控封包處理（可抓 NAT 改寫）
```bash
cilium monitor -t drop -t trace
```
* 會顯示每筆封包在 ingress / egress 被怎麼處理。
* 你可看到像是：
```
-> endpoint 2834 flow DROP: Policy denied ...
<- host flow FORWARDED: NAT 10.244.1.10 → 192.168.1.100:30000
```
---
### 🧪 三、測試指令（由 Pod 或節點測試）
#### 4. 從 Pod 發出封包
```bash
kubectl exec -it <pod> -- curl -v https://www.google.com
```
或直接 ping public IP:
```bash
kubectl exec -it <pod> -- ping 8.8.8.8
```
若 `ping` 通但 `curl` 不通，可能是 DNS 或 TLS 問題。
---
### 📡 四、確認路由與 egress 檢查
#### 5. 檢查節點的路由表
在出站節點查看：
```bash
ip route
```
* 確認是否有缺 default route（`default via ...`）
#### 6. 檢查 egress device 是否有 NAT
```bash
cilium status
```
* 確認 `masquerade` 是否啟用，例如：
```
Masquerading: BPF
```
---
### 🔧 五、常見故障點排查對照表
| 問題行為 | 原因推測 | 該查指令 |
| ------------------------ | ------------------------------- | ------------------------------------ |
| ping 外網失敗 | Pod 未 SNAT、Node 無 default route | `bpf nat list`, `ip route` |
| curl 外網失敗但 ping 正常 | DNS 問題、TLS 檢查失敗 | `cilium monitor`, `kubectl exec dig` |
| `bpf nat list` 無對應映射 | Cilium 未處理 NAT，或封包未走到 eBPF | 確認 `cilium status`、檢查 CNI 與配置 |
| conntrack 卡在 NEW/UNREPLY | 目的地有問題（防火牆/沒回來） | `bpf ct list` |
---
### 🛠️ 六、若需要手動清除 conntrack 或 NAT 映射（debug 用）
```bash
# 清除 conntrack table
cilium bpf ct flush global
# 清除 NAT table
cilium bpf nat flush global
```
---
如你能提供 Pod IP、節點 IP、目的地 IP（例如 8.8.8.8），我也可以協助你具體分析該封包是否已 NAT，是否進入 conntrack，是否 drop。是否要實際貼一段指令輸出來分析？
<br>
---
---
<br>
## 參考資料
### `kubectl -n kube-system exec ds/cilium -- cilium-dbg status --verbose`
```
$ kubectl -n kube-system exec ds/cilium -- cilium-dbg status --verbose
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init), install-cni-binaries (init)
KVStore: Disabled
Kubernetes: Ok 1.31 (v1.31.7) [linux/amd64]
Kubernetes APIs: ["EndpointSliceOrEndpoint", "cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumCIDRGroup", "core/v1::Namespace", "core/v1::Pods", "core/v1::Service", "networking.k8s.io/v1::NetworkPolicy"]
KubeProxyReplacement: True [bond1 10.78.26.99 (Direct Routing), bond0.206 10.78.26.238]
Host firewall: Disabled
SRv6: Disabled
CNI Chaining: none
CNI Config file: successfully wrote CNI configuration file to /host/etc/cni/net.d/05-cilium.conflist
Cilium: Ok 1.17.2 (v1.17.2-fb3ab54f)
NodeMonitor: Listening for events on 128 CPUs with 64x4096 of shared memory
Cilium health daemon: Ok
IPAM: IPv4: 14/254 allocated from 10.244.2.0/24,
Allocated addresses:
10.244.2.125 (default/vllm-add-vision3-mistral-0)
10.244.2.128 (gpu-operator/nvidia-device-plugin-daemonset-tvs4n)
10.244.2.189 (gpu-operator/gpu-feature-discovery-5fnfc)
10.244.2.222 (gpu-operator/nvidia-driver-daemonset-9j2sw)
10.244.2.225 (default/vllm-magistral-small-2506-0)
10.244.2.229 (slurm/slurm-compute-esc8000a-0)
10.244.2.233 (health)
10.244.2.235 (gpu-operator/nvidia-dcgm-exporter-8b2zp)
10.244.2.236 (gpu-operator/nvidia-container-toolkit-daemonset-fztrm)
10.244.2.51 (gpu-operator/nvidia-operator-validator-dm5q2)
10.244.2.62 (default/llamafactory-victor-657f946cfd-pkmcp)
10.244.2.64 (gpu-operator/gpu-operator-node-feature-discovery-worker-84ggc)
10.244.2.91 (router)
10.244.2.98 (slurm/slurm-compute-esc8000a-1)
IPv4 BIG TCP: Disabled
IPv6 BIG TCP: Disabled
BandwidthManager: Disabled
Routing: Network: Native Host: BPF
Attach Mode: Legacy TC
Device Mode: veth
Masquerading: BPF [bond1, bond0.206] 10.244.0.0/16 [IPv4: Enabled, IPv6: Disabled]
Clock Source for BPF: ktime
Controller Status: 81/81 healthy
Name Last success Last error Count Message
cilium-health-ep 42s ago never 0 no error
ct-map-pressure 10s ago never 0 no error
daemon-validate-config 33s ago never 0 no error
dns-garbage-collector-job 5s ago never 0 no error
endpoint-1013-regeneration-recovery never never 0 no error
endpoint-1022-regeneration-recovery never never 0 no error
endpoint-1156-regeneration-recovery never never 0 no error
endpoint-1226-regeneration-recovery never never 0 no error
endpoint-1233-regeneration-recovery never never 0 no error
endpoint-1585-regeneration-recovery never never 0 no error
endpoint-1729-regeneration-recovery never never 0 no error
endpoint-2097-regeneration-recovery never never 0 no error
endpoint-2340-regeneration-recovery never never 0 no error
endpoint-2866-regeneration-recovery never never 0 no error
endpoint-304-regeneration-recovery never never 0 no error
endpoint-3153-regeneration-recovery never never 0 no error
endpoint-427-regeneration-recovery never never 0 no error
endpoint-629-regeneration-recovery never never 0 no error
endpoint-gc 3m16s ago never 0 no error
endpoint-periodic-regeneration 32s ago never 0 no error
ep-bpf-prog-watchdog 25s ago never 0 no error
ipcache-inject-labels 29s ago never 0 no error
k8s-heartbeat 12s ago never 0 no error
link-cache 15s ago never 0 no error
node-neighbor-link-updater 6s ago 629h43m19s ago 0 no error
proxy-ports-checkpoint 629h43m29s ago never 0 no error
resolve-identity-1013 2m47s ago never 0 no error
resolve-identity-1022 2m57s ago never 0 no error
resolve-identity-1156 2m57s ago never 0 no error
resolve-identity-1226 3m24s ago never 0 no error
resolve-identity-1233 4m6s ago never 0 no error
resolve-identity-1585 3m24s ago never 0 no error
resolve-identity-1729 1m29s ago never 0 no error
resolve-identity-2097 4m49s ago never 0 no error
resolve-identity-2340 4m5s ago never 0 no error
resolve-identity-2866 3m23s ago never 0 no error
resolve-identity-304 1m3s ago never 0 no error
resolve-identity-3153 2m57s ago never 0 no error
resolve-identity-427 1m28s ago never 0 no error
resolve-identity-629 4m39s ago never 0 no error
resolve-labels-default/llamafactory-victor-657f946cfd-pkmcp 48h56m4s ago never 0 no error
resolve-labels-default/vllm-add-vision3-mistral-0 1m28s ago never 0 no error
resolve-labels-default/vllm-magistral-small-2506-0 1m29s ago never 0 no error
resolve-labels-gpu-operator/gpu-feature-discovery-5fnfc 629h38m2s ago never 0 no error
resolve-labels-gpu-operator/gpu-operator-node-feature-discovery-worker-84ggc 629h43m28s ago never 0 no error
resolve-labels-gpu-operator/nvidia-container-toolkit-daemonset-fztrm 629h39m54s ago never 0 no error
resolve-labels-gpu-operator/nvidia-dcgm-exporter-8b2zp 629h38m2s ago never 0 no error
resolve-labels-gpu-operator/nvidia-device-plugin-daemonset-tvs4n 629h38m1s ago never 0 no error
resolve-labels-gpu-operator/nvidia-driver-daemonset-9j2sw 629h43m28s ago never 0 no error
resolve-labels-gpu-operator/nvidia-operator-validator-dm5q2 629h37m52s ago never 0 no error
resolve-labels-slurm/slurm-compute-esc8000a-0 49h49m6s ago never 0 no error
resolve-labels-slurm/slurm-compute-esc8000a-1 49h49m5s ago never 0 no error
sync-lb-maps-with-k8s-services 629h43m29s ago never 0 no error
sync-policymap-1013 7m49s ago never 0 no error
sync-policymap-1022 7m59s ago never 0 no error
sync-policymap-1156 7m59s ago never 0 no error
sync-policymap-1226 13m23s ago never 0 no error
sync-policymap-1233 4m6s ago never 0 no error
sync-policymap-1585 13m23s ago never 0 no error
sync-policymap-1729 1m29s ago never 0 no error
sync-policymap-2097 9m51s ago never 0 no error
sync-policymap-2340 4m5s ago never 0 no error
sync-policymap-2866 13m23s ago never 0 no error
sync-policymap-304 11m3s ago never 0 no error
sync-policymap-3153 7m59s ago never 0 no error
sync-policymap-427 1m27s ago never 0 no error
sync-policymap-629 13m21s ago never 0 no error
sync-to-k8s-ciliumendpoint (1013) 6s ago never 0 no error
sync-to-k8s-ciliumendpoint (1022) 6s ago never 0 no error
sync-to-k8s-ciliumendpoint (1156) 6s ago never 0 no error
sync-to-k8s-ciliumendpoint (1226) 3s ago never 0 no error
sync-to-k8s-ciliumendpoint (1233) 2s ago never 0 no error
sync-to-k8s-ciliumendpoint (1585) 3s ago never 0 no error
sync-to-k8s-ciliumendpoint (1729) 9s ago never 0 no error
sync-to-k8s-ciliumendpoint (2097) 7s ago never 0 no error
sync-to-k8s-ciliumendpoint (2340) 11s ago never 0 no error
sync-to-k8s-ciliumendpoint (304) 10s ago never 0 no error
sync-to-k8s-ciliumendpoint (3153) 6s ago never 0 no error
sync-to-k8s-ciliumendpoint (427) 8s ago never 0 no error
sync-utime 42s ago never 0 no error
write-cni-file 629h43m31s ago never 0 no error
Proxy Status: OK, ip 10.244.2.91, 0 redirects active on ports 10000-20000, Envoy: external
Global Identity Range: min 256, max 65535
Hubble: Ok Current/Max Flows: 4095/4095 (100.00%), Flows/s: 14.08 Metrics: Disabled
KubeProxyReplacement Details:
Status: True
Socket LB: Enabled
Socket LB Tracing: Enabled
Socket LB Coverage: Full
Devices: bond1 10.78.26.99 (Direct Routing), bond0.206 10.78.26.238
Mode: SNAT
Backend Selection: Random
Session Affinity: Enabled
Graceful Termination: Enabled
NAT46/64 Support: Disabled
XDP Acceleration: Disabled
Services:
- ClusterIP: Enabled
- NodePort: Enabled (Range: 30000-32767)
- LoadBalancer: Enabled
- externalIPs: Enabled
- HostPort: Enabled
Annotations:
- service.cilium.io/node
- service.cilium.io/src-ranges-policy
- service.cilium.io/type
BPF Maps: dynamic sizing: on (ratio: 0.002500)
Name Size
Auth 524288
Non-TCP connection tracking 2371458
TCP connection tracking 4742916
Endpoint policy 65535
IP cache 512000
IPv4 masquerading agent 16384
IPv6 masquerading agent 16384
IPv4 fragmentation 8192
IPv4 service 65536
IPv6 service 65536
IPv4 service backend 65536
IPv6 service backend 65536
IPv4 service reverse NAT 65536
IPv6 service reverse NAT 65536
Metrics 1024
Ratelimit metrics 64
NAT 4742916
Neighbor table 4742916
Global policy 16384
Session affinity 65536
Sock reverse NAT 2371458
Tunnel 65536
Encryption: Disabled
Cluster health: 5/5 reachable (2025-07-18T03:25:38Z)
Name IP Node Endpoints
esc8000-01 (localhost):
Host connectivity to 10.78.26.99:
ICMP to stack: OK, RTT=148.205µs
HTTP to agent: OK, RTT=724.053µs
Endpoint connectivity to 10.244.2.233:
ICMP to stack: OK, RTT=218.766µs
HTTP to agent: OK, RTT=710.715µs
ocis-esc4000a-e11:
Host connectivity to 10.78.26.220:
ICMP to stack: OK, RTT=533.394µs
HTTP to agent: OK, RTT=866.409µs
Endpoint connectivity to 10.244.0.50:
ICMP to stack: OK, RTT=558.41µs
HTTP to agent: OK, RTT=1.002627ms
ocis-esc8000-e11p-02:
Host connectivity to 10.78.26.219:
ICMP to stack: OK, RTT=545.655µs
HTTP to agent: OK, RTT=884.727µs
Endpoint connectivity to 10.244.1.102:
ICMP to stack: OK, RTT=483.721µs
HTTP to agent: OK, RTT=868.475µs
ws-e900-g4-ws980t:
Host connectivity to 10.78.26.155:
ICMP to stack: OK, RTT=378.304µs
HTTP to agent: OK, RTT=2.333198ms
Endpoint connectivity to 10.244.4.92:
ICMP to stack: OK, RTT=312.289µs
HTTP to agent: OK, RTT=893.331µs
ws790:
Host connectivity to 10.78.26.76:
ICMP to stack: OK, RTT=526.5µs
HTTP to agent: OK, RTT=744.736µs
Endpoint connectivity to 10.244.3.1:
ICMP to stack: OK, RTT=615.678µs
HTTP to agent: OK, RTT=1.260051ms
Modules Health:
agent
├── controlplane
│ ├── auth
│ │ ├── observer-job-auth-gc-identity-events [OK] OK (2.581µs) [788] (89s, x1)
│ │ ├── observer-job-auth-request-authentication [OK] Primed (26d, x1)
│ │ └── timer-job-auth-gc-cleanup [OK] OK (19.743µs) (3m31s, x1)
│ ├── bgp-control-plane
│ │ └── job-diffstore-events [OK] Running (26d, x2)
│ ├── ciliumenvoyconfig
│ │ └── experimental
│ │ ├── job-reconcile [OK] OK, 0 object(s) (26d, x2)
│ │ └── job-refresh [OK] Next refresh in 30m0s (13m, x1260)
│ ├── daemon
│ │ ├── [OK] daemon-validate-config (33s, x37164)
│ │ ├── ep-bpf-prog-watchdog
│ │ │ └── ep-bpf-prog-watchdog [OK] ep-bpf-prog-watchdog (25s, x75559)
│ │ └── job-sync-hostips [OK] Synchronized (29s, x37785)
│ ├── dynamic-lifecycle-manager
│ │ ├── job-reconcile [OK] OK, 0 object(s) (43m, x631)
│ │ └── job-refresh [OK] Next refresh in 30m0s (13m, x1260)
│ ├── enabled-features
│ │ └── job-update-config-metric [OK] Waiting for agent config (26d, x1)
│ ├── endpoint-manager
│ │ ├── cilium-endpoint-1013 (gpu-operator/nvidia-operator-validator-dm5q2)
│ │ │ ├── cep-k8s-sync [OK] sync-to-k8s-ciliumendpoint (1013) (6s, x226651)
│ │ │ ├── datapath-regenerate [OK] Endpoint regeneration successful (32s, x18890)
│ │ │ └── policymap-sync [OK] sync-policymap-1013 (7m49s, x2519)
│ │ ├── cilium-endpoint-1022 (gpu-operator/gpu-feature-discovery-5fnfc)
│ │ │ ├── cep-k8s-sync [OK] sync-to-k8s-ciliumendpoint (1022) (6s, x226652)
│ │ │ ├── datapath-regenerate [OK] Endpoint regeneration successful (32s, x18890)
│ │ │ └── policymap-sync [OK] sync-policymap-1022 (7m59s, x2519)
│ │ ├── cilium-endpoint-1156 (gpu-operator/nvidia-dcgm-exporter-8b2zp)
│ │ │ ├── cep-k8s-sync [OK] sync-to-k8s-ciliumendpoint (1156) (6s, x226652)
│ │ │ ├── datapath-regenerate [OK] Endpoint regeneration successful (32s, x18890)
│ │ │ └── policymap-sync [OK] sync-policymap-1156 (7m59s, x2519)
│ │ ├── cilium-endpoint-1226 (gpu-operator/gpu-operator-node-feature-discovery-worker-84ggc)
│ │ │ ├── cep-k8s-sync [OK] sync-to-k8s-ciliumendpoint (1226) (3s, x226685)
│ │ │ ├── datapath-regenerate [OK] Endpoint regeneration successful (32s, x18894)
│ │ │ └── policymap-sync [OK] sync-policymap-1226 (13m, x2519)
│ │ ├── cilium-endpoint-123 (gpu-operator/nvidia-operator-validator-dm5q2)
│ │ │ └── cep-k8s-sync [OK] sync-to-k8s-ciliumendpoint (123) (26d, x1)
│ │ ├── cilium-endpoint-1233 (slurm/slurm-compute-esc8000a-0)
│ │ │ ├── cep-k8s-sync [OK] sync-to-k8s-ciliumendpoint (1233) (2s, x17935)
│ │ │ ├── datapath-regenerate [OK] Endpoint regeneration successful (32s, x1496)
│ │ │ └── policymap-sync [OK] sync-policymap-1233 (4m6s, x200)
│ │ ├── cilium-endpoint-1585 (gpu-operator/nvidia-driver-daemonset-9j2sw)
│ │ │ ├── cep-k8s-sync [OK] sync-to-k8s-ciliumendpoint (1585) (3s, x226685)
│ │ │ ├── datapath-regenerate [OK] Endpoint regeneration successful (32s, x18894)
│ │ │ └── policymap-sync [OK] sync-policymap-1585 (13m, x2519)
│ │ ├── cilium-endpoint-1729 (default/vllm-magistral-small-2506-0)
│ │ │ ├── cep-k8s-sync [OK] sync-to-k8s-ciliumendpoint (1729) (9s, x10)
│ │ │ ├── datapath-regenerate [OK] Endpoint regeneration successful (32s, x2)
│ │ │ └── policymap-sync [OK] sync-policymap-1729 (89s, x1)
│ │ ├── cilium-endpoint-2097 (gpu-operator/nvidia-container-toolkit-daemonset-fztrm)
│ │ │ ├── cep-k8s-sync [OK] sync-to-k8s-ciliumendpoint (2097) (7s, x226663)
│ │ │ ├── datapath-regenerate [OK] Endpoint regeneration successful (32s, x18891)
│ │ │ └── policymap-sync [OK] sync-policymap-2097 (9m51s, x2519)
│ │ ├── cilium-endpoint-2340 (slurm/slurm-compute-esc8000a-1)
│ │ │ ├── cep-k8s-sync [OK] sync-to-k8s-ciliumendpoint (2340) (1s, x17935)
│ │ │ ├── datapath-regenerate [OK] Endpoint regeneration successful (32s, x1496)
│ │ │ └── policymap-sync [OK] sync-policymap-2340 (4m5s, x200)
│ │ ├── cilium-endpoint-2866 (/)
│ │ │ ├── datapath-regenerate [OK] Endpoint regeneration successful (32s, x18893)
│ │ │ └── policymap-sync [OK] sync-policymap-2866 (13m, x2519)
│ │ ├── cilium-endpoint-304 (default/llamafactory-victor-657f946cfd-pkmcp)
│ │ │ ├── cep-k8s-sync [OK] sync-to-k8s-ciliumendpoint (304) (0s, x17617)
│ │ │ ├── datapath-regenerate [OK] Endpoint regeneration successful (32s, x1469)
│ │ │ └── policymap-sync [OK] sync-policymap-304 (11m, x196)
│ │ ├── cilium-endpoint-3153 (gpu-operator/nvidia-device-plugin-daemonset-tvs4n)
│ │ │ ├── cep-k8s-sync [OK] sync-to-k8s-ciliumendpoint (3153) (6s, x226652)
│ │ │ ├── datapath-regenerate [OK] Endpoint regeneration successful (32s, x18890)
│ │ │ └── policymap-sync [OK] sync-policymap-3153 (7m59s, x2519)
│ │ ├── cilium-endpoint-427 (default/vllm-add-vision3-mistral-0)
│ │ │ ├── cep-k8s-sync [OK] sync-to-k8s-ciliumendpoint (427) (8s, x10)
│ │ │ ├── datapath-regenerate [OK] Endpoint regeneration successful (32s, x2)
│ │ │ └── policymap-sync [OK] sync-policymap-427 (87s, x1)
│ │ ├── cilium-endpoint-629 (/)
│ │ │ ├── datapath-regenerate [OK] Endpoint regeneration successful (32s, x18903)
│ │ │ └── policymap-sync [OK] sync-policymap-629 (13m, x2519)
│ │ ├── cilium-endpoint-636 (slurm/slurm-compute-esc8000a-0)
│ │ │ └── cep-k8s-sync [OK] sync-to-k8s-ciliumendpoint (636) (3d1h, x1)
│ │ └── endpoint-gc [OK] endpoint-gc (3m16s, x7557)
│ ├── envoy-proxy
│ │ ├── observer-job-k8s-secrets-resource-events-cilium-secrets [OK] Primed (26d, x1)
│ │ └── timer-job-version-check [OK] OK (6.178771ms) (3m29s, x1)
│ ├── hubble
│ │ └── job-hubble [OK] Running (26d, x1)
│ ├── identity
│ │ └── timer-job-id-alloc-update-policy-maps [OK] OK (501.264µs) (88s, x1)
│ ├── l2-announcer
│ │ └── job-l2-announcer-lease-gc [OK] Running (26d, x1)
│ ├── nat-stats
│ │ └── timer-job-nat-stats [OK] OK (35.620643ms) (29s, x1)
│ ├── node-manager
│ │ ├── background-sync [OK] Node validation successful (74s, x21087)
│ │ ├── neighbor-link-updater
│ │ │ ├── ocis-esc4000a-e11 [OK] Node neighbor link update successful (6s, x21089)
│ │ │ ├── ocis-esc8000-e11p-02 [OK] Node neighbor link update successful (2m56s, x21094)
│ │ │ ├── ws-e900-g4-ws980t [OK] Node neighbor link update successful (66s, x21095)
│ │ │ └── ws790 [OK] Node neighbor link update successful (46s, x21154)
│ │ ├── node-checkpoint-writer [OK] node checkpoint written (9d, x58)
│ │ ├── nodes-add [OK] Node adds successful (26d, x5)
│ │ └── nodes-update [OK] Node updates successful (9d, x92)
│ ├── policy
│ │ └── observer-job-policy-importer [OK] OK (433.946µs) [102] (6d19h, x1)
│ ├── service-manager
│ │ ├── job-health-check-event-watcher [OK] Waiting for health check events (26d, x1)
│ │ └── job-service-reconciler [OK] 2 NodePort frontend addresses (26d, x1)
│ ├── service-resolver
│ │ └── job-service-reloader-initializer [OK] Running (26d, x1)
│ └── stale-endpoint-cleanup
│ └── job-endpoint-cleanup [OK] Running (26d, x1)
├── datapath
│ ├── agent-liveness-updater
│ │ └── timer-job-agent-liveness-updater [OK] OK (35.687µs) (0s, x1)
│ ├── iptables
│ │ ├── ipset
│ │ │ ├── job-ipset-init-finalizer [OK] Running (26d, x1)
│ │ │ ├── job-reconcile [OK] OK, 0 object(s) (43m, x632)
│ │ │ └── job-refresh [OK] Next refresh in 30m0s (13m, x1260)
│ │ └── job-iptables-reconciliation-loop [OK] iptables rules full reconciliation completed (4m4s, x1258)
│ ├── l2-responder
│ │ └── job-l2-responder-reconciler [OK] Running (26d, x1)
│ ├── maps
│ │ └── bwmap
│ │ └── timer-job-pressure-metric-throttle [OK] OK (3.393µs) (29s, x1)
│ ├── mtu
│ │ ├── job-endpoint-mtu-updater [OK] Endpoint MTU updated (26d, x1)
│ │ └── job-mtu-updater [OK] MTU updated (1500) (26d, x1)
│ ├── node-address
│ │ └── job-node-address-update [OK] 10.244.2.91 (primary) (26d, x1)
│ ├── orchestrator
│ │ └── job-reinitialize [OK] OK (26d, x2)
│ └── sysctl
│ ├── job-reconcile [OK] OK, 16 object(s) (3m19s, x114952)
│ └── job-refresh [OK] Next refresh in 9m51.521658506s (3m19s, x1)
└── infra
├── k8s-synced-crdsync
│ └── job-sync-crds [OK] Running (26d, x1)
├── metrics
│ ├── job-collect [OK] Sampled 24 metrics in 6.425202ms, next collection at 2025-07-18 03:28:32.639861649 +0000 UTC m=+2267102.776629908 (3m29s, x1)
│ └── timer-job-cleanup [OK] OK (198.082µs) (43m, x1)
└── shell
└── job-listener [OK] Listening on /var/run/cilium/shell.sock (26d, x1)
```
<br>
{%hackmd vaaMgNRPS4KGJDSFG0ZE0w %}
Sign in with Wallet
Connect another wallet