# OpenShift 設定讓不同的叢集節點中特定 namespace 底下所有 Pods 對外上網後顯示同一個 source ip ## 先備知識 **OpenShift 的 Egress IP 功能允許您確保一個或多個 Pod（來自一個或多個專案）的流量，在離開叢集網路時，透過 SNAT 機制，讓屬於特定 namespace 底下 Pods 的流量對外呈現固定 Source IP。** ## Step1: 建立 Project（Namespace）test-egress ``` oc new-project test-egress ``` ## Step2: 為特定節點指派託管 Egress IP 標記 worker-1 節點以託管 Egress IP 位址 ``` oc label nodes worker-1 k8s.ovn.org/egress-assignable="" ``` ## Step3: 為 Namespace 指派 Egress IP 1. 建立專案目錄 ``` mkdir ~/egressip ``` 1. 編輯與設定 EgressIP YAML File ``` echo 'apiVersion: k8s.ovn.org/v1 kind: EgressIP metadata: name: egressip-test spec: egressIPs: - 10.10.7.19 namespaceSelector: matchLabels: kubernetes.io/metadata.name: "test-egress"' > ~/egressip/egressip-test.yaml ``` > 1. `egressIPs`：列出要分配給 Pods 對外呈現的固定來源 IP。 > 2. `namespaceSelector`：以 Label Selector 的方式決定哪些 namespace 可套用此 Egress IP。 > - `kubernetes.io/metadata.name: "test"`，表示只選擇名稱為 `test-egress` 的 namesapce。 > 3. `podSelector`（可選）：可進一步針對特定 Pod 加以篩選；若不填，則預設會對該 namespace 內所有 Pods 生效。 1. 建立 EgressIP ``` oc apply -f ~/egressip/egressip-test.yaml ``` 1. 檢查 EgressIP 是否指派到節點上 ``` $ oc get egressip ``` 執行結果 : ``` NAME EGRESSIPS ASSIGNED NODE ASSIGNED EGRESSIPS egressip-test 10.10.7.19 worker-1 10.10.7.19 ``` 1. 實際到 worker-1 節點檢查 ``` $ ssh core@worker-1 'ip addr show | grep -B 6 10.10.7.19' ``` 執行結果 : ``` 5: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether bc:24:11:7f:f2:b0 brd ff:ff:ff:ff:ff:ff inet 10.10.7.16/24 brd 10.10.7.255 scope global noprefixroute br-ex valid_lft forever preferred_lft forever inet 169.254.0.2/17 brd 169.254.127.255 scope global br-ex valid_lft forever preferred_lft forever inet 10.10.7.19/32 scope global br-ex ``` ## Step4: 建立測試應用 1. 編輯 Deployment YAML File ``` echo 'apiVersion: apps/v1 kind: Deployment metadata: name: source-ip-app labels: app: source-ip-app spec: replicas: 3 selector: matchLabels: app: source-ip-app strategy: {} template: metadata: labels: app: source-ip-app spec: containers: - name: echoserver image: registry.k8s.io/echoserver:1.10 affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - worker-1 - worker-2 - worker-3 podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - source-ip-app topologyKey: "kubernetes.io/hostname"' > ~/egressip/source-ip-app.yaml ``` 1. 設定 Namespace 的 default ServiceAccount 使用 anyuid SCC ``` oc adm policy add-scc-to-user anyuid -z default -n test-egress ``` 1. 建立測試應用 ``` oc apply -f ~/egressip/source-ip-app.yaml ``` 1. 檢視應用 Pods 是否正常 ``` oc get pods -o wide ``` 執行結果 : ``` NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES source-ip-app-b44758d6d-t894p 1/1 Running 0 3s 10.128.2.27 worker-1 <none> <none> source-ip-app-b44758d6d-tzpjj 1/1 Running 0 3s 10.131.0.14 worker-2 <none> <none> source-ip-app-b44758d6d-xvsqg 1/1 Running 0 4s 10.130.0.9 worker-3 <none> <none> ``` ## Step5: 在 OpenShift 叢集外部的主機建立外部 HTTP server 1. 安裝 httpd 套建 ``` sudo yum install -y httpd ``` 1. 將 httpd 啟在 `8080` Port ``` sudo sed -i 's|Listen 80|Listen 8080|g' /etc/httpd/conf/httpd.conf ``` 1. 建立測試 HTML 頁面 ``` echo '<html> <head/> <body>OK</body> </html>' | sudo tee /var/www/html/index.html ``` 1. 啟動 httpd Server 服務 ``` sudo systemctl start httpd ``` 1. 確認 httpd Server 服務運作正常 ``` sudo systemctl is-active httpd ``` 正確執行結果 : ``` active ``` 1. 測試頁面是否正常運作 ``` curl http://localhost:8080/index.html ``` 執行結果 : ``` <html> <head/> <body>OK</body> </html> ``` ## Step6: 測試連線 1. 撰寫 Bash Script 來完成以下目的 : 1. 以 round-robin 的方式對名稱為 `source-ip-app-...` 前綴的 Pod 執行 `curl http://10.10.7.11:8080/index.html` 2. 編輯 Bash Script ``` nano ~/egressip/test.sh ``` 檔案內容如下 : ``` #!/bin/bash mapfile -t pods < <(oc -n test-egress get pods -o name | grep source-ip-app-) if [[ ${#pods[@]} -eq 0 ]]; then echo "找不到符合條件的 Pod，請確認 Pod 名稱前綴是否正確" >&2 exit 1 fi for i in {1..20}; do for pod in "${pods[@]}"; do pod_name="${pod#pod/}" echo "對 Pod ${pod_name} 執行 curl 請求..." oc exec "${pod_name}" -- curl -s http://10.10.7.11:8080/index.html \ && echo " ==> ${pod_name} 執行成功" \ || echo " ==> ${pod_name} 執行失敗" sleep 1 done done ``` 1. 為 Bash Script 賦予執行權限 ``` chmod +x ~/egressip/test.sh ``` 1. 執行腳本開始測試 ``` ~/egressip/test.sh ``` 執行結果 : ``` 對 Pod source-ip-app-b44758d6d-t894p 執行 curl 請求... <html> <head/> <body>OK</body> </html> ==> source-ip-app-b44758d6d-t894p 執行成功 對 Pod source-ip-app-b44758d6d-tzpjj 執行 curl 請求... <html> <head/> <body>OK</body> </html> ==> source-ip-app-b44758d6d-tzpjj 執行成功 對 Pod source-ip-app-b44758d6d-xvsqg 執行 curl 請求... ...以下省略 ``` 1. 檢視 httpd access log ``` sudo tail -f /var/log/httpd/access_log ``` 執行結果 : ``` 127.0.0.1 - - [05/Jun/2025:16:51:26 +0800] "GET /index.html HTTP/1.1" 200 39 "-" "curl/7.76.1" 10.10.7.19 - - [05/Jun/2025:17:01:56 +0800] "GET /index.html HTTP/1.1" 200 39 "-" "curl/7.47.0" 10.10.7.19 - - [05/Jun/2025:17:01:57 +0800] "GET /index.html HTTP/1.1" 200 39 "-" "curl/7.47.0" 10.10.7.19 - - [05/Jun/2025:17:01:59 +0800] "GET /index.html HTTP/1.1" 200 39 "-" "curl/7.47.0" 10.10.7.19 - - [05/Jun/2025:17:02:00 +0800] "GET /index.html HTTP/1.1" 200 39 "-" "curl/7.47.0" 10.10.7.19 - - [05/Jun/2025:17:02:02 +0800] "GET /index.html HTTP/1.1" 200 39 "-" "curl/7.47.0" 10.10.7.19 - - [05/Jun/2025:17:02:04 +0800] "GET /index.html HTTP/1.1" 200 39 "-" "curl/7.47.0" 10.10.7.19 - - [05/Jun/2025:17:02:05 +0800] "GET /index.html HTTP/1.1" 200 39 "-" "curl/7.47.0" 10.10.7.19 - - [05/Jun/2025:17:02:07 +0800] "GET /index.html HTTP/1.1" 200 39 "-" "curl/7.47.0" 10.10.7.19 - - [05/Jun/2025:17:02:08 +0800] "GET /index.html HTTP/1.1" 200 39 "-" "curl/7.47.0" 10.10.7.19 - - [05/Jun/2025:17:02:10 +0800] "GET /index.html HTTP/1.1" 200 39 "-" "curl/7.47.0" 10.10.7.19 - - [05/Jun/2025:17:02:11 +0800] "GET /index.html HTTP/1.1" 200 39 "-" "curl/7.47.0" 10.10.7.19 - - [05/Jun/2025:17:02:13 +0800] "GET /index.html HTTP/1.1" 200 39 "-" "curl/7.47.0" 10.10.7.19 - - [05/Jun/2025:17:02:15 +0800] "GET /index.html HTTP/1.1" 200 39 "-" "curl/7.47.0" 10.10.7.19 - - [05/Jun/2025:17:02:16 +0800] "GET /index.html HTTP/1.1" 200 39 "-" "curl/7.47.0" ...以下結果省略 ``` ## Step7: 清理環境 ``` oc delete -R -f ~/egressip/ oc label node worker-1 k8s.ovn.org/egress-assignable- oc adm policy remove-scc-from-user anyuid -z default -n test-egress oc delete project test-egress # 以下命令請在 httpd Server 的主機上執行 sudo systemcctl stop httpd sudo yum remove -y httpd ``` ## 參考資料 - [https://docs.redhat.com/en/documentation/openshift_container_platform/4.18/html/networking/ovn-kubernetes-network-plugin#configuring-egress-ips-ovn](https://docs.redhat.com/en/documentation/openshift_container_platform/4.18/html/networking/ovn-kubernetes-network-plugin#configuring-egress-ips-ovn)