# 在全新安装的 K8s 中安装 Cilium 之后会导致 Pod 无法访问互联网和本地网络

## 故障

- 无法从 Pod 内访问 10.0/16
- 无法从 Pod 内访问互联网
- 可以从 Pod 内访问集群节点

## 用于集群安装的配置

sudo cat kubeadm.yml

apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: <token>
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 10.24.0.2 # 作为 control plane 的节点 1 的 IP
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///run/containerd/containerd.sock
  imagePullPolicy: IfNotPresent
  name: node1
  taints: null
---
apiServer:
  timeoutForControlPlane: 20m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.k8s.io
kind: ClusterConfiguration
kubernetesVersion: 1.28.0
networking:
  podSubnet: 10.244.0.0/16
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
scheduler: {}
cgroupDriver: systemd

## 用于执行 `sudo cilium install --version 1.14.2 --values cilium-values.yml --dry-run-helm-values > cilium-values-initial.yaml` 时使用的预配置

sudo cat cilium-values.yml

hubble:
  relay:
    enabled: true
  ui:
    enabled: true

ipam:
  mode: 'kubernetes'
  operator:
    clusterPoolIPv4PodCIDRList: '10.244.0.0/16'

ipv4NativeRoutingCIDR: '10.244.0.0/16'
enableIPv4Masquerade: false
enableIPv6Masquerade: false
autoDirectNodeRoutes: true
tunnel: disabled

## 用于 `sudo helm install cilium cilium/cilium --namespace kube-system --values cilium-values-initial.yaml` 安装 Cilium 时使用的配置

sudo cat cilium-values-initial.yaml

autoDirectNodeRoutes: true
cluster:
  name: kubernetes
enableIPv4Masquerade: false
enableIPv6Masquerade: false
hubble:
  relay:
    enabled: true
  ui:
    enabled: true
ipam:
  mode: kubernetes
  operator:
    clusterPoolIPv4PodCIDRList: 10.244.0.0/16
ipv4NativeRoutingCIDR: 10.244.0.0/16
k8sServiceHost: 10.24.0.2
k8sServicePort: 6443
kubeProxyReplacement: strict
operator:
  replicas: 1
serviceAccounts:
  cilium:
    name: cilium
  operator:
    name: cilium-operator
tunnel: disabled

## 使用 curl 排查

如果使用 `sudo kubectl run -it --rm test --image=curlimages/curl --restart=Never -- /bin/sh` 排查，会发现网络请求被挂起。

## 参考资料

[Cannot reach external endpoint with service ip when the external endpoint is one of k8s node · Issue #16235 · cilium/cilium](https://github.com/cilium/cilium/issues/16235)

[kubernetes - Can't access the external network from pod (nginx-pod) - Stack Overflow](https://stackoverflow.com/questions/76432743/cant-access-the-external-network-from-pod-nginx-pod)

[linux - Kubernetes Nodes are not reachable and cannot reach local network after installing cilium - Server Fault](https://serverfault.com/questions/1103034/kubernetes-nodes-are-not-reachable-and-cannot-reach-local-network-after-installi)

[Pod cannot access external network · Issue #20085 · cilium/cilium](https://github.com/cilium/cilium/issues/20085)

[CI: ConformanceAKS: curl succeeded while it should have failed due to incorrect exit code · Issue #22162 · cilium/cilium](https://github.com/cilium/cilium/issues/22162)

[cilium connectivity test failures · Issue #673 · cilium/cilium-cli](https://github.com/cilium/cilium-cli/issues/673)

[Cilium deployment fails to pass conn test and sonobuoy · Issue #8546 · kubernetes-sigs/kubespray](https://github.com/kubernetes-sigs/kubespray/issues/8546)