NGAP Load Balancing with LoxiLB and free5GC

# NGAP Load Balancing with LoxiLB and free5GC (L7 load-balancer) Based on [NGAP Load Balancing with LoxiLB](https://www.loxilb.io/post/ngap-load-balancing-with-loxilb), but change 5G Core to use **free5gc** instead of Open5gs # Overall Architecture (Single-Cluster) * Single Kubernetes Cluster: All Free5GC network functions run in a single K8s cluster * LoxiLB: LoxiLB will within that same cluster to load-balance traffic destined for the AMF’s N2 interface (SCTP). # Preparing LoxiLB instance LoxiLB runs as a standalone Docker container (--net=host). ``` $ apt-get update $ apt-get install -y software-properties-common $ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - $ add-apt-repository -y "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" $ apt-get update $ apt-get install -y docker-ce // additional option fallback to prevent dns be affected $ docker run -u root --cap-add SYS_ADMIN --restart unless-stopped --privileged \ -dit -v /dev/log:/dev/log --entrypoint=/root/loxilb-io/loxilb/loxilb \ --net=host --name loxilb ghcr.io/loxilb-io/loxilb:scp --fallback ``` # Deploy kube-loxilb 1. **loxiURL**: the IP address of the machine running the LoxiLB data-plane and to talk to the external LoxiLB instance. `Host network IP: 192.168.56.106 (enp0s8)` 2. **cidrPools**: ``` args: - --loxiURL=http://192.168.56.106:11111 - --cidrPools=defaultPool=10.100.50.249/32 #- --monitor #- --setBGP=64512 #- --extBGPPeers=50.50.50.1:65101,51.51.51.1:65102 #- --setRoles - --setLBMode=2 #- --config=/opt/loxilb/agent/kube-loxilb.conf ``` deploy kube-loxilb. ``` vboxuser@test100:~/work/l7lb$ kubectl apply -f kube-loxilb.yaml serviceaccount/kube-loxilb created clusterrole.rbac.authorization.k8s.io/kube-loxilb created clusterrolebinding.rbac.authorization.k8s.io/kube-loxilb created deployment.apps/kube-loxilb created ``` verify that the Deployment has been created and listening ``` vboxuser@test100:~/work/l7lb$ kubectl -n kube-system get deployment NAME READY UP-TO-DATE AVAILABLE AGE calico-kube-controllers 1/1 1 1 62d coredns 1/1 1 1 62d kube-loxilb 1/1 1 1 4h45m vboxuser@test1000:~/work/free5gc-helm$ kubectl get pod -n kube-system NAME READY STATUS RESTARTS AGE calico-kube-controllers-77bd7c5b-2fmzs 1/1 Running 17 (10m ago) 21d calico-node-fmlj7 1/1 Running 10 (10m ago) 21d coredns-864597b5fd-pk5p8 1/1 Running 1 (10m ago) 12m kube-loxilb-7bb475df98-h5xx2 1/1 Running 6 (10m ago) 20d kube-multus-ds-5j2k4 1/1 Running 9 (10m ago) 21d # server is running vboxuser@test100:~/work/l7lb$ curl http://192.168.56.106:11111/ {"code":404,"message":"path / was not found"} ``` ``` docker exec loxilb loxicmd get lb -o wide ``` # Deploy free5gC using helm Modify the AMF service to use LoadBalancer type and add LoxiLB annotations `free5gc/values.yaml` ```yaml global: amf: n2if: # NGAP ipAddress: 192.168.56.103 # Change to LoxiLB IP service: ngap: enabled: true name: amf-n2 port: 38412 protocol: SCTP type: LoadBalancer annotations: loxilb.io/probetype: "none" loxilb.io/lbmode: "fullproxy" loxilb.io/epselect: "n2" ``` ## Deploy free5gC ``` helm install -n free5gc free5gc-helm ./free5gc/ \ --set global.n6network.masterIf=enp0s8 \ --set global.n6network.subnetIP="192.168.56.0" \ --set global.n6network.gatewayIP="192.168.56.1" \ --set free5gc-upf.upf1.n6if.ipAddress="192.168.56.200" \ --set free5gc-upf.upf2.n6if.ipAddress="192.168.56.201" \ --set free5gc-upf.upfb.n6if.ipAddress="192.168.56.202" \ --set global.n2network.masterIf=enp0s8 \ --set global.n3network.masterIf=enp0s8 \ --set global.n4network.masterIf=enp0s8 \ --set global.n9network.masterIf=enp0s8 helm install -n free5gc ueransim ./ueransim/ \ --set global.n2network.masterIf=enp0s8 \ --set global.n3network.masterIf=enp0s8 helm install -n free5gc ueransim-1 ./ueransim/ \ --set global.n2network.masterIf=enp0s8 \ --set global.n3network.masterIf=enp0s8 helm install -n free5gc ueransim-2 ./ueransim2/ \ --set global.n2network.masterIf=enp0s8 \ --set global.n3network.masterIf=enp0s8 helm uninstall -n free5gc free5gc-helm kubectl delete pv free5gc-pv-cert kubectl delete pvc datadir-mongodb-0 -n free5gc kubectl delete pv free5gc-mongodb-pv kubectl apply -f persistent-vol-for-mongodb.yaml kubectl apply -f persistent-vol-for-cert.yaml helm uninstall -n free5gc ueransim kubectl get pv,pvc -n free5gc kubectl get all -n free5gc ``` # Deploy kube-loxilb v2 use: LoxiLB address: 10.100.50.249 ``` args: - --loxiURL=http://192.168.56.106:11111 - --cidrPools=defaultPool=10.100.50.249/32 - --setLBMode=2 ``` ## Deploy free5gC using helm Modify the AMF service to use LoadBalancer type and add LoxiLB annotations ``` ------------------------------- amf: n2if: # NGAP ipAddress: 10.100.50.249 service: ngap: enabled: true name: amf-n2 port: 38412 protocol: SCTP type: LoadBalancer loadBalancerClass: "loxilb.io/loxilb" # annotations: loxilb.io/probetype: "none" loxilb.io/lbmode: "fullproxy" loxilb.io/epselect: "n2" loxilb.io/lbclass: "loxilb.io/loxilb" ``` ## Verify the services at LoxiLB: ``` vboxuser@test1000:~/work/free5gc-helm$ docker exec -it loxilb loxicmd get lb -o wide | EXT IP | SEC IPS | HOST | PORT | PROTO | NAME | MARK | SEL | MODE | ENDPOINT | EPORT | WEIGHT | STATE | COUNTERS | |---------------|---------|------|-------|-------|---------------------------------------------------|------|-----|-----------|-----------|-------|--------|-------|----------| | 10.100.50.249 | | | 38412 | sctp | free5gc_free5gc-helm-free5gc-amf-amf-n2:llb-inst0 | 0 | n2 | fullproxy | 10.0.2.15 | 31412 | 1 | - | 0:0 | ``` ``` vboxuser@test1000:~/repo/free5gc-helm$ kubectl logs -n kube-system kube-loxilb-7bb475df98-h5xx2 I0206 08:30:35.599345 1 logLevel.go:40] Set log level to 0 I0206 08:30:35.701801 1 agent.go:65] Starting kube-loxilb: I0206 08:30:35.701828 1 agent.go:66] Version: I0206 08:30:35.701833 1 agent.go:67] Build: 2025_01_21_00h:39m-main-b39d568 I0206 08:30:35.702873 1 client.go:41] No kubeconfig file was specified. Falling back to in-cluster config I0206 08:30:35.807811 1 agent.go:98] URLs: [http://192.168.56.106:11111] I0206 08:30:35.807841 1 agent.go:99] LB Class: loxilb.io/loxilb I0206 08:30:35.807847 1 agent.go:100] CIDR Pools: [defaultPool=10.100.50.249/32] I0206 08:30:35.807853 1 agent.go:101] CIDR6 Pools: [defaultPool=3ffe:cafe::1/96] I0206 08:30:35.807857 1 agent.go:102] SetBGP: 0 I0206 08:30:35.807861 1 agent.go:103] ListenBGPPort: 179 I0206 08:30:35.807865 1 agent.go:104] eBGPMultiHop: false I0206 08:30:35.807869 1 agent.go:105] SetLBMode: 2 I0206 08:30:35.807874 1 agent.go:106] ExclIPAM: false I0206 08:30:35.807879 1 agent.go:107] Monitor: false I0206 08:30:35.807883 1 agent.go:108] ExtBGPPeers: [] I0206 08:30:35.807888 1 agent.go:109] SetRoles: I0206 08:30:35.807919 1 agent.go:110] Zone: llb I0206 08:30:35.807927 1 agent.go:111] Zone instances 1 I0206 08:30:35.807932 1 agent.go:112] AppendEPs: false I0206 08:30:35.807957 1 agent.go:156] created external IP Pool (CIDR: defaultPool:10.100.50.249/32) I0206 08:30:35.808231 1 agent.go:180] created external IP6 Pool (CIDR: defaultPool:3ffe:cafe::1/96) I0206 08:30:35.808282 1 client.go:126] NewLoxiClient Created: http://192.168.56.106:11111 I0206 08:30:35.812072 1 loadbalancer.go:330] Starting LoxilbLoadBalancerManager I0206 08:30:35.812440 1 loxiurl.go:177] Starting LoxiLBURLManager I0206 08:30:35.813824 1 shared_informer.go:313] Waiting for caches to sync for LoxilbLoadBalancerManager I0206 08:30:35.899749 1 client.go:186] LoxiHealthCheckChan: loxilb-lb(192.168.56.106) is alive I0206 08:30:35.899804 1 agent.go:271] Client alive : 192.168.56.106 I0206 08:30:36.002624 1 egress.go:158] Starting LoxilbEgressManager I0206 08:30:36.798954 1 shared_informer.go:320] Caches are synced for LoxilbLoadBalancerManager I0206 08:30:36.800064 1 loadbalancer.go:756] New LbCache free5gc/free5gc-helm-free5gc-amf-amf-n2 Added I0206 08:30:36.800114 1 loadbalancer.go:1008] free5gc/free5gc-helm-free5gc-amf-amf-n2: Added(false) Update(true) needDelete(false) I0206 08:30:36.800131 1 loadbalancer.go:1009] Endpoint IP Pairs [10.0.2.15] I0206 08:30:36.800150 1 loadbalancer.go:1010] Secondary IP Pairs [] I0206 08:30:36.815181 1 loadbalancer.go:1309] loxilb-lb(192.168.56.106): add lb {{10.100.50.249 38412 sctp 5 4 false false 1800 0 true none 0 0 0 0 free5gc_free5gc-helm-free5gc-amf-amf-n2:llb-inst0 0 false false} [] [] [{10.0.2.15 31412 1 }]} vboxuser@test1000:~/repo/free5gc-helm$ ``` ## run three ue to check loadbalancer ``` kubectl logs -n free5gc -f deployment/ueransim-gnb kubectl logs -n free5gc -f deployment/ueransim-ue ``` ### check ue work ``` vboxuser@test1000:~/work/free5gc-helm/charts$ kubectl exec -it -n free5gc deployment/ueransim-ue -- ping -i uesimtun0 8.8.8.8 ping: option argument contains garbage: uesimtun0 ping: this will become fatal error in the future PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data. 64 bytes from 8.8.8.8: icmp_seq=1 ttl=254 time=150 ms 64 bytes from 8.8.8.8: icmp_seq=2 ttl=254 time=136 ms 64 bytes from 8.8.8.8: icmp_seq=3 ttl=254 time=117 ms 64 bytes from 8.8.8.8: icmp_seq=5 ttl=254 time=82.1 ms 64 bytes from 8.8.8.8: icmp_seq=4 ttl=254 time=101 ms 64 bytes from 8.8.8.8: icmp_seq=6 ttl=254 time=68.4 ms 64 bytes from 8.8.8.8: icmp_seq=7 ttl=254 time=48.7 ms 64 bytes from 8.8.8.8: icmp_seq=8 ttl=254 time=36.6 ms 64 bytes from 8.8.8.8: icmp_seq=9 ttl=254 time=17.7 ms ^C --- 8.8.8.8 ping statistics --- 38 packets transmitted, 9 received, 76.3158% packet loss, time 605ms rtt min/avg/max/mdev = 17.661/84.102/149.627/42.859 ms, pipe 10, ipg/ewma 16.348/93.905 ms vboxuser@test1000:~/work/free5gc-helm/charts$ kubectl exec -it -n free5gc deployment/free5gc-helm-free5gc-upf-upf1 -- ip route default dev n6 scope link 10.1.0.0/17 dev upfgtp proto static 10.1.128.0/17 dev upfgtp proto static 10.100.50.224/29 dev n9 proto kernel scope link src 10.100.50.226 10.100.50.240/29 dev n4 proto kernel scope link src 10.100.50.242 169.254.1.1 dev eth0 scope link 192.168.56.0/24 dev n6 proto kernel scope link src 192.168.56.200 vboxuser@test1000:~/work/free5gc-helm/charts$ ``` ULCL have three upf, CHECK whether ue connect diff node ``` pod/free5gc-helm-free5gc-upf-upf1-58fbf87ff-whjsf 1/1 Running 0 83s pod/free5gc-helm-free5gc-upf-upf2-6bddf9969f-slpb7 1/1 Running 0 83s pod/free5gc-helm-free5gc-upf-upfb-6b8cdb8fcc-tpqfd 1/1 Running 0 83s ```

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.