NGAP Load Balancing with LoxiLB and free5GC (L7 load-balancer)

Based on NGAP Load Balancing with LoxiLB, but change 5G Core to use free5gc instead of Open5gs

Overall Architecture (Single-Cluster)

Single Kubernetes Cluster: All Free5GC network functions run in a single K8s cluster
LoxiLB: LoxiLB will within that same cluster to load-balance traffic destined for the AMF’s N2 interface (SCTP).

Preparing LoxiLB instance

LoxiLB runs as a standalone Docker container (–net=host).

$ apt-get update
$ apt-get install -y software-properties-common
$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
$ add-apt-repository -y "deb [arch=amd64] https://download.docker.com/linux/ubuntu  $(lsb_release -cs)  stable"
$ apt-get update
$ apt-get install -y docker-ce


// additional option fallback to prevent dns be affected
$ docker run -u root --cap-add SYS_ADMIN --restart unless-stopped --privileged \
   -dit -v /dev/log:/dev/log --entrypoint=/root/loxilb-io/loxilb/loxilb \
   --net=host --name loxilb ghcr.io/loxilb-io/loxilb:scp --fallback

Deploy kube-loxilb

loxiURL: the IP address of the machine running the LoxiLB data-plane and to talk to the external LoxiLB instance.
Host network IP: 192.168.56.106 (enp0s8)
cidrPools:

        args:
        - --loxiURL=http://192.168.56.106:11111
        - --cidrPools=defaultPool=10.100.50.249/32
        #- --monitor
        #- --setBGP=64512
        #- --extBGPPeers=50.50.50.1:65101,51.51.51.1:65102
        #- --setRoles
        - --setLBMode=2
        #- --config=/opt/loxilb/agent/kube-loxilb.conf

deploy kube-loxilb.

vboxuser@test100:~/work/l7lb$ kubectl apply -f kube-loxilb.yaml
serviceaccount/kube-loxilb created
clusterrole.rbac.authorization.k8s.io/kube-loxilb created
clusterrolebinding.rbac.authorization.k8s.io/kube-loxilb created
deployment.apps/kube-loxilb created

verify that the Deployment has been created and listening

vboxuser@test100:~/work/l7lb$ kubectl -n kube-system get deployment
NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
calico-kube-controllers   1/1     1            1           62d
coredns                   1/1     1            1           62d
kube-loxilb               1/1     1            1           4h45m

vboxuser@test1000:~/work/free5gc-helm$ kubectl get pod -n kube-system
NAME                                     READY   STATUS    RESTARTS       AGE
calico-kube-controllers-77bd7c5b-2fmzs   1/1     Running   17 (10m ago)   21d
calico-node-fmlj7                        1/1     Running   10 (10m ago)   21d
coredns-864597b5fd-pk5p8                 1/1     Running   1 (10m ago)    12m
kube-loxilb-7bb475df98-h5xx2             1/1     Running   6 (10m ago)    20d
kube-multus-ds-5j2k4                     1/1     Running   9 (10m ago)    21d

# server is running
vboxuser@test100:~/work/l7lb$ curl http://192.168.56.106:11111/
{"code":404,"message":"path / was not found"}

docker exec loxilb loxicmd get lb -o wide

Deploy free5gC using helm

Modify the AMF service to use LoadBalancer type and add LoxiLB annotations

free5gc/values.yaml

global:
  amf:
    n2if:  # NGAP
      ipAddress: 192.168.56.103  # Change to LoxiLB IP
    service:
      ngap:
        enabled: true
        name: amf-n2
        port: 38412
        protocol: SCTP
        type: LoadBalancer
        annotations:
          loxilb.io/probetype: "none"
          loxilb.io/lbmode: "fullproxy"
          loxilb.io/epselect: "n2"

Deploy free5gC

helm install -n free5gc free5gc-helm ./free5gc/ \
--set global.n6network.masterIf=enp0s8 \
--set global.n6network.subnetIP="192.168.56.0" \
--set global.n6network.gatewayIP="192.168.56.1" \
--set free5gc-upf.upf1.n6if.ipAddress="192.168.56.200" \
--set free5gc-upf.upf2.n6if.ipAddress="192.168.56.201" \
--set free5gc-upf.upfb.n6if.ipAddress="192.168.56.202" \
--set global.n2network.masterIf=enp0s8 \
--set global.n3network.masterIf=enp0s8 \
--set global.n4network.masterIf=enp0s8 \
--set global.n9network.masterIf=enp0s8

helm install -n free5gc ueransim ./ueransim/ \
--set global.n2network.masterIf=enp0s8 \
--set global.n3network.masterIf=enp0s8

helm install -n free5gc ueransim-1 ./ueransim/ \
--set global.n2network.masterIf=enp0s8 \
--set global.n3network.masterIf=enp0s8

helm install -n free5gc ueransim-2 ./ueransim2/ \
--set global.n2network.masterIf=enp0s8 \
--set global.n3network.masterIf=enp0s8

helm uninstall -n free5gc free5gc-helm
kubectl delete pv free5gc-pv-cert
kubectl delete pvc datadir-mongodb-0 -n free5gc
kubectl delete pv free5gc-mongodb-pv
kubectl apply -f persistent-vol-for-mongodb.yaml
kubectl apply -f persistent-vol-for-cert.yaml

helm uninstall -n free5gc ueransim

kubectl get pv,pvc -n free5gc

kubectl get all -n free5gc

Deploy kube-loxilb v2

use:
LoxiLB address: 10.100.50.249

args:
- --loxiURL=http://192.168.56.106:11111
- --cidrPools=defaultPool=10.100.50.249/32
- --setLBMode=2

Deploy free5gC using helm

Modify the AMF service to use LoadBalancer type and add LoxiLB annotations

-------------------------------
  amf:
    n2if:  # NGAP
      ipAddress: 10.100.50.249
    service:
      ngap:
        enabled: true
        name: amf-n2
        port: 38412
        protocol: SCTP
        type: LoadBalancer
        loadBalancerClass: "loxilb.io/loxilb"  # 
        annotations:
          loxilb.io/probetype: "none"
          loxilb.io/lbmode: "fullproxy"
          loxilb.io/epselect: "n2"
          loxilb.io/lbclass: "loxilb.io/loxilb"

Verify the services at LoxiLB:

vboxuser@test1000:~/work/free5gc-helm$ docker exec -it loxilb loxicmd get lb -o wide
|    EXT IP     | SEC IPS | HOST | PORT  | PROTO |                       NAME                        | MARK | SEL |   MODE    | ENDPOINT  | EPORT | WEIGHT | STATE | COUNTERS |
|---------------|---------|------|-------|-------|---------------------------------------------------|------|-----|-----------|-----------|-------|--------|-------|----------|
| 10.100.50.249 |         |      | 38412 | sctp  | free5gc_free5gc-helm-free5gc-amf-amf-n2:llb-inst0 |    0 | n2  | fullproxy | 10.0.2.15 | 31412 |      1 | -     | 0:0      |

vboxuser@test1000:~/repo/free5gc-helm$ kubectl logs -n kube-system kube-loxilb-7bb475df98-h5xx2
I0206 08:30:35.599345       1 logLevel.go:40] Set log level to 0
I0206 08:30:35.701801       1 agent.go:65] Starting kube-loxilb:
I0206 08:30:35.701828       1 agent.go:66]   Version: 
I0206 08:30:35.701833       1 agent.go:67]   Build: 2025_01_21_00h:39m-main-b39d568
I0206 08:30:35.702873       1 client.go:41] No kubeconfig file was specified. Falling back to in-cluster config
I0206 08:30:35.807811       1 agent.go:98] URLs: [http://192.168.56.106:11111]
I0206 08:30:35.807841       1 agent.go:99] LB Class: loxilb.io/loxilb
I0206 08:30:35.807847       1 agent.go:100] CIDR Pools: [defaultPool=10.100.50.249/32]
I0206 08:30:35.807853       1 agent.go:101] CIDR6 Pools: [defaultPool=3ffe:cafe::1/96]
I0206 08:30:35.807857       1 agent.go:102] SetBGP: 0
I0206 08:30:35.807861       1 agent.go:103] ListenBGPPort: 179
I0206 08:30:35.807865       1 agent.go:104] eBGPMultiHop: false
I0206 08:30:35.807869       1 agent.go:105] SetLBMode: 2
I0206 08:30:35.807874       1 agent.go:106] ExclIPAM: false
I0206 08:30:35.807879       1 agent.go:107] Monitor: false
I0206 08:30:35.807883       1 agent.go:108] ExtBGPPeers: []
I0206 08:30:35.807888       1 agent.go:109] SetRoles: 
I0206 08:30:35.807919       1 agent.go:110] Zone: llb
I0206 08:30:35.807927       1 agent.go:111] Zone instances 1
I0206 08:30:35.807932       1 agent.go:112] AppendEPs: false
I0206 08:30:35.807957       1 agent.go:156] created external IP Pool (CIDR: defaultPool:10.100.50.249/32)
I0206 08:30:35.808231       1 agent.go:180] created external IP6 Pool (CIDR: defaultPool:3ffe:cafe::1/96)
I0206 08:30:35.808282       1 client.go:126] NewLoxiClient Created: http://192.168.56.106:11111
I0206 08:30:35.812072       1 loadbalancer.go:330] Starting LoxilbLoadBalancerManager
I0206 08:30:35.812440       1 loxiurl.go:177] Starting LoxiLBURLManager
I0206 08:30:35.813824       1 shared_informer.go:313] Waiting for caches to sync for LoxilbLoadBalancerManager
I0206 08:30:35.899749       1 client.go:186] LoxiHealthCheckChan: loxilb-lb(192.168.56.106) is alive
I0206 08:30:35.899804       1 agent.go:271] Client alive : 192.168.56.106
I0206 08:30:36.002624       1 egress.go:158] Starting LoxilbEgressManager
I0206 08:30:36.798954       1 shared_informer.go:320] Caches are synced for LoxilbLoadBalancerManager
I0206 08:30:36.800064       1 loadbalancer.go:756] New LbCache free5gc/free5gc-helm-free5gc-amf-amf-n2 Added
I0206 08:30:36.800114       1 loadbalancer.go:1008] free5gc/free5gc-helm-free5gc-amf-amf-n2: Added(false) Update(true) needDelete(false)
I0206 08:30:36.800131       1 loadbalancer.go:1009] Endpoint IP Pairs [10.0.2.15]
I0206 08:30:36.800150       1 loadbalancer.go:1010] Secondary IP Pairs []
I0206 08:30:36.815181       1 loadbalancer.go:1309] loxilb-lb(192.168.56.106): add lb {{10.100.50.249  38412 sctp 5 4 false false 1800 0 true none 0   0 0 0 free5gc_free5gc-helm-free5gc-amf-amf-n2:llb-inst0 0  false false} [] [] [{10.0.2.15 31412 1  }]}
vboxuser@test1000:~/repo/free5gc-helm$

run three ue to check loadbalancer

kubectl logs -n free5gc -f deployment/ueransim-gnb
kubectl logs -n free5gc -f deployment/ueransim-ue

check ue work

vboxuser@test1000:~/work/free5gc-helm/charts$ kubectl exec -it -n free5gc deployment/ueransim-ue -- ping -i uesimtun0 8.8.8.8 
ping: option argument contains garbage: uesimtun0 
ping: this will become fatal error in the future 
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data. 
64 bytes from 8.8.8.8: icmp_seq=1 ttl=254 time=150 ms 
64 bytes from 8.8.8.8: icmp_seq=2 ttl=254 time=136 ms 
64 bytes from 8.8.8.8: icmp_seq=3 ttl=254 time=117 ms 
64 bytes from 8.8.8.8: icmp_seq=5 ttl=254 time=82.1 ms 
64 bytes from 8.8.8.8: icmp_seq=4 ttl=254 time=101 ms 
64 bytes from 8.8.8.8: icmp_seq=6 ttl=254 time=68.4 ms 
64 bytes from 8.8.8.8: icmp_seq=7 ttl=254 time=48.7 ms 
64 bytes from 8.8.8.8: icmp_seq=8 ttl=254 time=36.6 ms 
64 bytes from 8.8.8.8: icmp_seq=9 ttl=254 time=17.7 ms 
^C 
--- 8.8.8.8 ping statistics --- 
38 packets transmitted, 9 received, 76.3158% packet loss, time 605ms 
rtt min/avg/max/mdev = 17.661/84.102/149.627/42.859 ms, pipe 10, ipg/ewma 16.348/93.905 ms 
vboxuser@test1000:~/work/free5gc-helm/charts$ kubectl exec -it -n free5gc deployment/free5gc-helm-free5gc-upf-upf1 -- ip route 
default dev n6 scope link  
10.1.0.0/17 dev upfgtp proto static  
10.1.128.0/17 dev upfgtp proto static  
10.100.50.224/29 dev n9 proto kernel scope link src 10.100.50.226  
10.100.50.240/29 dev n4 proto kernel scope link src 10.100.50.242  
169.254.1.1 dev eth0 scope link  
192.168.56.0/24 dev n6 proto kernel scope link src 192.168.56.200  
vboxuser@test1000:~/work/free5gc-helm/charts$

ULCL have three upf, CHECK whether ue connect diff node

pod/free5gc-helm-free5gc-upf-upf1-58fbf87ff-whjsf            1/1     Running    0          83s
pod/free5gc-helm-free5gc-upf-upf2-6bddf9969f-slpb7           1/1     Running    0          83s
pod/free5gc-helm-free5gc-upf-upfb-6b8cdb8fcc-tpqfd           1/1     Running    0          83s

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`	在筆記中貼入程式碼
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.