---
# System prepended metadata

title: CiliumNetworkPolicy 實戰
tags: [K8S]

---

# CiliumNetworkPolicy 實戰

CiliumNetworkPolicy 是 Cilium 提供的一種網路策略資源，用於 Kubernetes 中針對細微的網路流量控制和安全性管理。相比 Kubernetes 的原生 NetworkPolicy，CiliumNetworkPolicy 提供了更多高級功能，特別是針對 L3（IP）、L4（port）、L7（應用層協議）的網路流量規則。


## 實作

* 已部署好 rke2 1m2w,CNI: cilium

### Layer 3 Examples
* 環境準備
```
$ kubectl create ns demo

$ echo 'apiVersion: v1
kind: Pod
metadata:
  name: frontend
  namespace: demo
  labels:
    role: frontend
spec:
  containers:
  - name: nginx
    image: nginx:latest
    ports:
    - containerPort: 80' | kubectl apply -f -
    
$ echo 'apiVersion: v1
kind: Pod
metadata:
  name: backend
  namespace: demo
  labels:
    role: backend
spec:
  containers:
  - name: nginx
    image: nginx:latest
    ports:
    - containerPort: 80' | kubectl apply -f -
    
$ echo 'apiVersion: v1
kind: Pod
metadata:
  name: other
  namespace: demo
  labels:
    role: other
spec:
  containers:
  - name: nginx
    image: nginx:latest
    ports:
    - containerPort: 80' | kubectl apply -f -
```
* 查看標籤
```
$ kubectl -n demo get po --show-labels
NAME       READY   STATUS    RESTARTS   AGE   LABELS
backend    1/1     Running   0          45s   role=backend
frontend   1/1     Running   0          54s   role=frontend
other      1/1     Running   0          37s   role=other
```
* 僅允許具有標籤 `role: frontend` 的 Pod 發送流量到 `role: backend` 的 Pod，沒有帶這個標籤都不能夠訪問到 `role: frontend` 的 Pod。
```
$ echo 'apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "l3-rule"
  namespace: demo
spec:
  endpointSelector:
    matchLabels:
      role: backend
  ingress:
  - fromEndpoints:
    - matchLabels:
        role: frontend' | kubectl apply -f -

$ kubectl -n demo get cnp
NAME      AGE
l3-rule   3s
```


* 驗證可以透過從 frontend 訪問到 backend，但是 other 沒辦法訪問 backend。
```
$ kubectl -n demo get po -owide
NAME       READY   STATUS    RESTARTS   AGE     IP            NODE        NOMINATED NODE   READINESS GATES
backend    1/1     Running   0          34s     10.42.3.54    cilium-w1   <none>           <none>
frontend   1/1     Running   0          4m57s   10.42.1.89    cilium-m3   <none>           <none>
other      1/1     Running   0          4m40s   10.42.1.226   cilium-m3   <none>           <none>


$ kubectl -n demo exec frontend -- curl 10.42.3.54
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>


$ kubectl -n demo exec other -- curl 10.42.3.54
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:03 --:--:--     0

# 連從本機都不能夠訪問
# curl 10.42.3.54
^C
```

* 環境清除
```
$ kubectl -n demo delete pod/backend pod/frontend pod/other cnp/l3-rule
```

### Layer 4 Examples


* 環境準備
```
$ echo 'apiVersion: v1
kind: Pod
metadata:
  name: frontend
  namespace: demo
  labels:
    app: myservice
spec:
  containers:
  - name: nginx
    image: nginx:latest
    ports:
    - containerPort: 80' | kubectl apply -f -
    
$ echo 'apiVersion: v1
kind: Pod
metadata:
  name: backend
  namespace: demo
  labels:
    role: myservice
spec:
  containers:
  - name: nginx
    image: nginx:latest
    ports:
    - containerPort: 80' | kubectl apply -f -

$ echo 'apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    role: other
  name: other
  namespace: demo
spec:
  containers:
  - image: quay.io/cloudwalker/busybox
    name: bx
    command:
      - /bin/sh
      - -c
      - |
        mkdir www
        echo hi > www/index.html
        busybox httpd -p 9000 -h www -f
    ports:
    - containerPort: 9000' | kubectl apply -f -
```
```
$ kubectl -n demo get po -owide
NAME       READY   STATUS    RESTARTS   AGE     IP            NODE        NOMINATED NODE   READINESS GATES
backend    1/1     Running   0          4m32s   10.42.3.179   cilium-w1   <none>           <none>
frontend   1/1     Running   0          4m39s   10.42.1.114   cilium-m3   <none>           <none>
other      1/1     Running   0          10s     10.42.2.13    cilium-m2   <none>           <none>
```

* 限制 demo namespace 下所有標籤為 `app: myservice` 的 Pod 的 Egress（出去） 的流量，僅允許這些 Pod 對外部發送 TCP 協議的 80 port。
```
$ echo 'apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "l4-rule"
  namespace: demo
spec:
  endpointSelector:
    matchLabels:
      app: myservice
  egress:
    - toPorts:
      - ports:
        - port: "80"
          protocol: TCP' | kubectl apply -f -

$ kubectl -n demo get cnp
NAME      AGE
l4-rule   7s
```


* 驗證 frontend 可以透過 80 port 訪問到 backend，但是無法透過 9000 port 訪問到 other。
```
$ kubectl -n demo exec frontend -- curl 10.42.3.179:80
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>

$ kubectl -n demo exec frontend -- curl 10.42.2.13:9000
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:03 --:--:--     0
```

* 環境清除
```
$ kubectl -n demo delete pod/backend pod/frontend pod/other cnp/l4-rule
```

### Layer 7 Examples
* 環境準備
```
$ echo 'apiVersion: v1
kind: Pod
metadata:
  name: frontend
  namespace: demo
  labels:
    env: prod
spec:
  containers:
  - name: nginx
    image: nginx:latest
    ports:
    - containerPort: 80' | kubectl apply -f -

$ echo 'kind: Pod
apiVersion: v1
metadata:
  name: backend
  namespace: demo
  labels:
    app: service
spec:
  containers:
  - command:
    - /agnhost
    - netexec
    - --http-port
    - "8080"
    image: registry.k8s.io/e2e-test-images/agnhost:2.39
    name: backend' | kubectl apply -f -
```
* 預設 frontend 都可以使用 POST,GET 等 api request 訪問 backend。
```
$ kubectl -n demo get po -owide
NAME       READY   STATUS    RESTARTS   AGE   IP           NODE        NOMINATED NODE   READINESS GATES
backend    1/1     Running   0          6s    10.42.3.60   cilium-w1   <none>           <none>
frontend   1/1     Running   0          15s   10.42.1.9    cilium-m3   <none>           <none>


$ kubectl -n demo exec frontend -- curl -s -v -X GET 10.42.3.60:8080/hostname
*   Trying 10.42.3.60:8080...
* Connected to 10.42.3.60 (10.42.3.60) port 8080 (#0)
> GET /hostname HTTP/1.1
> Host: 10.42.3.60:8080
> User-Agent: curl/7.88.1
> Accept: */*
>
backend< HTTP/1.1 200 OK

$ kubectl -n demo exec frontend -- curl -s -v -X POST 10.42.3.60:8080/hostname
*   Trying 10.42.3.60:8080...
* Connected to 10.42.3.60 (10.42.3.60) port 8080 (#0)
> POST /hostname HTTP/1.1
> Host: 10.42.3.60:8080
> User-Agent: curl/7.88.1
> Accept: */*
>
backend< HTTP/1.1 200 OK
```
* 僅允許 HTTP GET 透過 8080 port 訪問路徑 "/hostname"，其他方法（如 POST、PUT）或路徑將被拒絕。
```
$ echo 'apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "l7-rule"
  namespace: demo
spec:
  description: "Allow HTTP GET /hostname from env=prod to app=service"
  endpointSelector:
    matchLabels:
      app: service
  ingress:
  - fromEndpoints:
    - matchLabels:
        env: prod
    toPorts:
    - ports:
      - port: "8080"
        protocol: TCP
      rules:
        http:
        - method: "GET"
          path: "/hostname"' | kubectl apply -f -
```

* 驗證 frontend 只能透過 GET request 訪問 "/hostname" 位置，而透過 POST request 訪問就被拒絕了。
```
$ kubectl -n demo exec frontend -- curl -s -v -X GET 10.42.3.60:8080/hostname
*   Trying 10.42.3.60:8080...
* Connected to 10.42.3.60 (10.42.3.60) port 8080 (#0)
> GET /hostname HTTP/1.1
> Host: 10.42.3.60:8080
> User-Agent: curl/7.88.1
> Accept: */*
>
backend< HTTP/1.1 200 OK


$ kubectl -n demo exec frontend -- curl -s -v -X POST 10.42.3.60:8080/hostname
*   Trying 10.42.3.60:8080...
* Connected to 10.42.3.60 (10.42.3.60) port 8080 (#0)
> POST /hostname HTTP/1.1
> Host: 10.42.3.60:8080
> User-Agent: curl/7.88.1
> Accept: */*
>
Access denied    # 請求被拒絕了
```
* 環境清除
```
$ kubectl -n demo delete pod/backend pod/frontend cnp/l7-rule
```

## mTLS pod 雙向加密
* cilium 需要啟用以下功能才有 mTLS 功能
```
$ cilium install \
    --set authentication.mutual.spire.enabled=true \
    --set authentication.mutual.spire.install.enabled=true
```

* 環境準備
```
$ kubectl create namespace nodebb
$ kubectl -n nodebb create deployment nodebb --image=quay.io/flysangel/library/nginx
$ kubectl -n nodebb create deployment redis --image=quay.io/flysangel/library/redis
$ kubectl -n nodebb create service nodeport nodebb --tcp=80:80 --node-port=30000
```
* 首先，在 nodebb 命名空間中建立一個名為 nodebb 的 L4 CiliumNetworkPolicy，並進行如下配置：允許在命名空間 ingress-nginx 中運行的所有 Pod 存取 nodebb Deployment 的 Pod。需要相互認證。
* 接下來，將上一步建立的網路策略擴充如下：允許主機存取nodebb Deployment的Pods。不要使用相互認證。
* 實作
```
$ curl -s -o /dev/null -w "%{http_code}\n" \
--connect-timeout 2 http://k8s.local/nodebb
200

$ nano cilium1.yaml
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: nodebb
  namespace: nodebb
spec:
  endpointSelector:
    matchLabels:
      app: nodebb
  ingress:
  - fromEndpoints:
    - matchLabels:
        "k8s:io.kubernetes.pod.namespace": ingress-nginx
    authentication:
      mode: "required"
    toPorts:
    - ports:
      - port: "80"
        protocol: TCP
  - fromEntities:
    - host
    toPorts:
    - ports:
      - port: "80"
        protocol: TCP


$ kubectl apply -f cilium1.yaml
```
* 驗證
```
$ kubectl run test --image=quay.io/flysangel/alpine:base-v1.0.2
$ kubectl get pod test
NAME   READY   STATUS    RESTARTS   AGE
test   1/1     Running   0          12s

$ extip=$(kubectl -n ingress-nginx get service ingress-nginx-controller -o jsonpath='{.spec.loadBalancerIP}')
$ kubectl exec -it test -- curl -s -o /dev/null \
-w "%{http_code}\n" \
--resolve k8s.local:80:"${extip}" \
--connect-timeout 2 http://k8s.local/nodebb
200

# 允許從主機訪問
$ curl -s -o /dev/null -w "%{http_code}\n" \
http://localhost:30000
200

$ nodeip=$(kubectl get node -o jsonpath='{.items[*].status.addresses[?(@.type=="InternalIP")].address}')
$ kubectl exec -it test -- curl -s -o /dev/null -w "%{http_code}\n" \
--connect-timeout 2 \
http://${nodeip}:30000
000
command terminated with exit code 28
```
## 參考

https://docs.cilium.io/en/stable/network/kubernetes/policy/#networkpolicy