# CiliumNetworkPolicy 實戰 CiliumNetworkPolicy 是 Cilium 提供的一種網路策略資源，用於 Kubernetes 中針對細微的網路流量控制和安全性管理。相比 Kubernetes 的原生 NetworkPolicy，CiliumNetworkPolicy 提供了更多高級功能，特別是針對 L3（IP）、L4（port）、L7（應用層協議）的網路流量規則。 ## 實作 * 已部署好 rke2 1m2w,CNI: cilium ### Layer 3 Examples * 環境準備 ``` $ kubectl create ns demo $ echo 'apiVersion: v1 kind: Pod metadata: name: frontend namespace: demo labels: role: frontend spec: containers: - name: nginx image: nginx:latest ports: - containerPort: 80' | kubectl apply -f - $ echo 'apiVersion: v1 kind: Pod metadata: name: backend namespace: demo labels: role: backend spec: containers: - name: nginx image: nginx:latest ports: - containerPort: 80' | kubectl apply -f - $ echo 'apiVersion: v1 kind: Pod metadata: name: other namespace: demo labels: role: other spec: containers: - name: nginx image: nginx:latest ports: - containerPort: 80' | kubectl apply -f - ``` * 查看標籤 ``` $ kubectl -n demo get po --show-labels NAME READY STATUS RESTARTS AGE LABELS backend 1/1 Running 0 45s role=backend frontend 1/1 Running 0 54s role=frontend other 1/1 Running 0 37s role=other ``` * 僅允許具有標籤 `role: frontend` 的 Pod 發送流量到 `role: backend` 的 Pod，沒有帶這個標籤都不能夠訪問到 `role: frontend` 的 Pod。 ``` $ echo 'apiVersion: "cilium.io/v2" kind: CiliumNetworkPolicy metadata: name: "l3-rule" namespace: demo spec: endpointSelector: matchLabels: role: backend ingress: - fromEndpoints: - matchLabels: role: frontend' | kubectl apply -f - $ kubectl -n demo get cnp NAME AGE l3-rule 3s ``` * 驗證可以透過從 frontend 訪問到 backend，但是 other 沒辦法訪問 backend。 ``` $ kubectl -n demo get po -owide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES backend 1/1 Running 0 34s 10.42.3.54 cilium-w1 <none> <none> frontend 1/1 Running 0 4m57s 10.42.1.89 cilium-m3 <none> <none> other 1/1 Running 0 4m40s 10.42.1.226 cilium-m3 <none> <none> $ kubectl -n demo exec frontend -- curl 10.42.3.54 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0<!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> $ kubectl -n demo exec other -- curl 10.42.3.54 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0 # 連從本機都不能夠訪問 # curl 10.42.3.54 ^C ``` * 環境清除 ``` $ kubectl -n demo delete pod/backend pod/frontend pod/other cnp/l3-rule ``` ### Layer 4 Examples * 環境準備 ``` $ echo 'apiVersion: v1 kind: Pod metadata: name: frontend namespace: demo labels: app: myservice spec: containers: - name: nginx image: nginx:latest ports: - containerPort: 80' | kubectl apply -f - $ echo 'apiVersion: v1 kind: Pod metadata: name: backend namespace: demo labels: role: myservice spec: containers: - name: nginx image: nginx:latest ports: - containerPort: 80' | kubectl apply -f - $ echo 'apiVersion: v1 kind: Pod metadata: creationTimestamp: null labels: role: other name: other namespace: demo spec: containers: - image: quay.io/cloudwalker/busybox name: bx command: - /bin/sh - -c - | mkdir www echo hi > www/index.html busybox httpd -p 9000 -h www -f ports: - containerPort: 9000' | kubectl apply -f - ``` ``` $ kubectl -n demo get po -owide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES backend 1/1 Running 0 4m32s 10.42.3.179 cilium-w1 <none> <none> frontend 1/1 Running 0 4m39s 10.42.1.114 cilium-m3 <none> <none> other 1/1 Running 0 10s 10.42.2.13 cilium-m2 <none> <none> ``` * 限制 demo namespace 下所有標籤為 `app: myservice` 的 Pod 的 Egress（出去） 的流量，僅允許這些 Pod 對外部發送 TCP 協議的 80 port。 ``` $ echo 'apiVersion: "cilium.io/v2" kind: CiliumNetworkPolicy metadata: name: "l4-rule" namespace: demo spec: endpointSelector: matchLabels: app: myservice egress: - toPorts: - ports: - port: "80" protocol: TCP' | kubectl apply -f - $ kubectl -n demo get cnp NAME AGE l4-rule 7s ``` * 驗證 frontend 可以透過 80 port 訪問到 backend，但是無法透過 9000 port 訪問到 other。 ``` $ kubectl -n demo exec frontend -- curl 10.42.3.179:80 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0<!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> $ kubectl -n demo exec frontend -- curl 10.42.2.13:9000 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0 ``` * 環境清除 ``` $ kubectl -n demo delete pod/backend pod/frontend pod/other cnp/l4-rule ``` ### Layer 7 Examples * 環境準備 ``` $ echo 'apiVersion: v1 kind: Pod metadata: name: frontend namespace: demo labels: env: prod spec: containers: - name: nginx image: nginx:latest ports: - containerPort: 80' | kubectl apply -f - $ echo 'kind: Pod apiVersion: v1 metadata: name: backend namespace: demo labels: app: service spec: containers: - command: - /agnhost - netexec - --http-port - "8080" image: registry.k8s.io/e2e-test-images/agnhost:2.39 name: backend' | kubectl apply -f - ``` * 預設 frontend 都可以使用 POST,GET 等 api request 訪問 backend。 ``` $ kubectl -n demo get po -owide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES backend 1/1 Running 0 6s 10.42.3.60 cilium-w1 <none> <none> frontend 1/1 Running 0 15s 10.42.1.9 cilium-m3 <none> <none> $ kubectl -n demo exec frontend -- curl -s -v -X GET 10.42.3.60:8080/hostname * Trying 10.42.3.60:8080... * Connected to 10.42.3.60 (10.42.3.60) port 8080 (#0) > GET /hostname HTTP/1.1 > Host: 10.42.3.60:8080 > User-Agent: curl/7.88.1 > Accept: */* > backend< HTTP/1.1 200 OK $ kubectl -n demo exec frontend -- curl -s -v -X POST 10.42.3.60:8080/hostname * Trying 10.42.3.60:8080... * Connected to 10.42.3.60 (10.42.3.60) port 8080 (#0) > POST /hostname HTTP/1.1 > Host: 10.42.3.60:8080 > User-Agent: curl/7.88.1 > Accept: */* > backend< HTTP/1.1 200 OK ``` * 僅允許 HTTP GET 透過 8080 port 訪問路徑 "/hostname"，其他方法（如 POST、PUT）或路徑將被拒絕。 ``` $ echo 'apiVersion: "cilium.io/v2" kind: CiliumNetworkPolicy metadata: name: "l7-rule" namespace: demo spec: description: "Allow HTTP GET /hostname from env=prod to app=service" endpointSelector: matchLabels: app: service ingress: - fromEndpoints: - matchLabels: env: prod toPorts: - ports: - port: "8080" protocol: TCP rules: http: - method: "GET" path: "/hostname"' | kubectl apply -f - ``` * 驗證 frontend 只能透過 GET request 訪問 "/hostname" 位置，而透過 POST request 訪問就被拒絕了。 ``` $ kubectl -n demo exec frontend -- curl -s -v -X GET 10.42.3.60:8080/hostname * Trying 10.42.3.60:8080... * Connected to 10.42.3.60 (10.42.3.60) port 8080 (#0) > GET /hostname HTTP/1.1 > Host: 10.42.3.60:8080 > User-Agent: curl/7.88.1 > Accept: */* > backend< HTTP/1.1 200 OK $ kubectl -n demo exec frontend -- curl -s -v -X POST 10.42.3.60:8080/hostname * Trying 10.42.3.60:8080... * Connected to 10.42.3.60 (10.42.3.60) port 8080 (#0) > POST /hostname HTTP/1.1 > Host: 10.42.3.60:8080 > User-Agent: curl/7.88.1 > Accept: */* > Access denied # 請求被拒絕了 ``` * 環境清除 ``` $ kubectl -n demo delete pod/backend pod/frontend cnp/l7-rule ``` ## mTLS pod 雙向加密 * cilium 需要啟用以下功能才有 mTLS 功能 ``` $ cilium install \ --set authentication.mutual.spire.enabled=true \ --set authentication.mutual.spire.install.enabled=true ``` * 環境準備 ``` $ kubectl create namespace nodebb $ kubectl -n nodebb create deployment nodebb --image=quay.io/flysangel/library/nginx $ kubectl -n nodebb create deployment redis --image=quay.io/flysangel/library/redis $ kubectl -n nodebb create service nodeport nodebb --tcp=80:80 --node-port=30000 ``` * 首先，在 nodebb 命名空間中建立一個名為 nodebb 的 L4 CiliumNetworkPolicy，並進行如下配置：允許在命名空間 ingress-nginx 中運行的所有 Pod 存取 nodebb Deployment 的 Pod。需要相互認證。 * 接下來，將上一步建立的網路策略擴充如下：允許主機存取nodebb Deployment的Pods。不要使用相互認證。 * 實作 ``` $ curl -s -o /dev/null -w "%{http_code}\n" \ --connect-timeout 2 http://k8s.local/nodebb 200 $ nano cilium1.yaml apiVersion: "cilium.io/v2" kind: CiliumNetworkPolicy metadata: name: nodebb namespace: nodebb spec: endpointSelector: matchLabels: app: nodebb ingress: - fromEndpoints: - matchLabels: "k8s:io.kubernetes.pod.namespace": ingress-nginx authentication: mode: "required" toPorts: - ports: - port: "80" protocol: TCP - fromEntities: - host toPorts: - ports: - port: "80" protocol: TCP $ kubectl apply -f cilium1.yaml ``` * 驗證 ``` $ kubectl run test --image=quay.io/flysangel/alpine:base-v1.0.2 $ kubectl get pod test NAME READY STATUS RESTARTS AGE test 1/1 Running 0 12s $ extip=$(kubectl -n ingress-nginx get service ingress-nginx-controller -o jsonpath='{.spec.loadBalancerIP}') $ kubectl exec -it test -- curl -s -o /dev/null \ -w "%{http_code}\n" \ --resolve k8s.local:80:"${extip}" \ --connect-timeout 2 http://k8s.local/nodebb 200 # 允許從主機訪問 $ curl -s -o /dev/null -w "%{http_code}\n" \ http://localhost:30000 200 $ nodeip=$(kubectl get node -o jsonpath='{.items[*].status.addresses[?(@.type=="InternalIP")].address}') $ kubectl exec -it test -- curl -s -o /dev/null -w "%{http_code}\n" \ --connect-timeout 2 \ http://${nodeip}:30000 000 command terminated with exit code 28 ``` ## 參考 https://docs.cilium.io/en/stable/network/kubernetes/policy/#networkpolicy