Try   HackMD

NGAP Load Balancing with LoxiLB and free5GC (L7 load-balancer)

Based on NGAP Load Balancing with LoxiLB and 5g-sctp-loadbalancer, but change 5G Core to use free5gc instead of Open5gs

This implementation is primarily based on above two references

setup and target:

I'm trying to implement SCTP load balancing for Free5GC AMF using LoxiLB, but encountering connection issues. Here's my setup:

Environment:

  • Single Kubernetes cluster running Free5GC components
  • LoxiLB running as standalone Docker container
  • Two AMF instances
  • UERANSIM for testing

Configuration:

  1. LoxiLB VIP: 10.100.50.249
  2. AMF1: 10.100.50.251
  3. AMF2: 10.100.50.252

Overall Architecture (Single-Cluster)

  • Single Kubernetes Cluster: All Free5GC network functions run in a single K8s cluster
  • LoxiLB: LoxiLB will within that same cluster to load-balance traffic destined for the AMF’s N2 interface (SCTP).

Preparing docker LoxiLB instance

LoxiLB runs as a standalone Docker container.

docker run -u root --cap-add SYS_ADMIN --restart unless-stopped --privileged -dit -v /dev/log:/dev/log --entrypoint=/root/loxilb-io/loxilb/loxilb --net=host --name loxilb ghcr.io/loxilb-io/loxilb:scp --proxyonlymode

Deploy kube-loxilb

  1. loxiURL: the IP address of the machine running the LoxiLB data-plane and to talk to the external LoxiLB instance.
    Host network IP: 192.168.56.106 (enp0s8)
        args:
        - --loxiURL=http://192.168.56.106:11111
        - --cidrPools=defaultPool=10.100.50.249/32
        #- --monitor
        #- --setBGP=64512
        #- --extBGPPeers=50.50.50.1:65101,51.51.51.1:65102
        #- --setRoles
        - --setLBMode=2
        #- --config=/opt/loxilb/agent/kube-loxilb.conf

kubectl apply -f kube-loxilb.yaml

verify status

vboxuser@test1000:~/work/free5gc-helm/charts$ docker ps
CONTAINER ID   IMAGE                          COMMAND                  CREATED        STATUS             PORTS     NAMES
5b1a2ef6d7e0   ghcr.io/loxilb-io/loxilb:scp   "/root/loxilb-io/lox…"   41 hours ago   Up About an hour             loxilb
vboxuser@test1000:~/work/free5gc-helm/charts$ kubectl get pod -n kube-system
NAME                                     READY   STATUS    RESTARTS       AGE
calico-kube-controllers-77bd7c5b-2fmzs   1/1     Running   30 (25h ago)   27d
calico-node-jhvc4                        1/1     Running   12 (25h ago)   5d11h
coredns-5cf7b49c88-5vwcz                 1/1     Running   10 (25h ago)   5d5h
kube-loxilb-7bb475df98-s8zrb             1/1     Running   0              38m
kube-multus-ds-5j2k4                     1/1     Running   22 (25h ago)   27d

Deploy free5gC and ueransim using helm

Modify the AMF service

Modify the AMF service to use LoadBalancer type and add LoxiLB annotations

free5gc/values.yaml

change:












  amf:
    n2if:  # NGAP
      ipAddress: 10.100.50.250
    service:
      ngap:
        enabled: false
        name: amf-n2
        port: 38412
        nodeport: 31412
        protocol: SCTP
        type: NodePort

to:

































  amf:
    n2if:  # NGAP
      ipAddress: 10.100.50.251
    service:
      ngap:
        enabled: true
        name: amf-n2
        port: 38412
        protocol: SCTP
        type: LoadBalancer
        loadBalancerClass: "loxilb.io/loxilb"  # 
        annotations:
          loxilb.io/probetype: "none"
          loxilb.io/lbmode: "fullproxy"
          loxilb.io/epselect: "n2"
          loxilb.io/lbclass: "loxilb.io/loxilb" 
  amf2:
    n2if:  # NGAP
      ipAddress: 10.100.50.252
    service:
      ngap:
        enabled: true
        name: amf2-n2
        port: 38412
        protocol: SCTP
        type: LoadBalancer
        loadBalancerClass: "loxilb.io/loxilb"  # 
        annotations:
          loxilb.io/probetype: "none"
          loxilb.io/lbmode: "fullproxy"
          loxilb.io/epselect: "n2"
          loxilb.io/lbclass: "loxilb.io/loxilb"

create amf2 folder to apply amf2

then run:

helm install -n free5gc free5gc-helm ./free5gc/ \
--set global.n6network.masterIf=enp0s8 \
--set global.n6network.subnetIP="192.168.56.0" \
--set global.n6network.gatewayIP="192.168.56.1" \
--set free5gc-upf.upf1.n6if.ipAddress="192.168.56.200" \
--set free5gc-upf.upf2.n6if.ipAddress="192.168.56.201" \
--set free5gc-upf.upfb.n6if.ipAddress="192.168.56.202" \
--set global.n2network.masterIf=enp0s8 \
--set global.n3network.masterIf=enp0s8 \
--set global.n4network.masterIf=enp0s8 \
--set global.n9network.masterIf=enp0s8

verify the free5gc and ueransim

see many problem

vboxuser@test1000:~/work/free5gc-helm/charts$ kubectl get all -n free5gc 
NAME                                                         READY   STATUS     RESTARTS   AGE
pod/free5gc-helm-free5gc-amf-amf-66ddf68dc9-rjtht            1/1     Running    0          81m
pod/free5gc-helm-free5gc-amf2-amf2-687bd6879b-zdj68          1/1     Running    0          81m
pod/free5gc-helm-free5gc-ausf-ausf-64c684f546-5psjf          1/1     Running    0          81m
pod/free5gc-helm-free5gc-chf-chf-7c7bb88fb7-89lwh            1/1     Running    0          81m
pod/free5gc-helm-free5gc-dbpython-dbpython-59684d749-6pkv6   1/1     Running    0          81m
pod/free5gc-helm-free5gc-nef-nef-759b6dfbdb-w5fwm            1/1     Running    0          81m
pod/free5gc-helm-free5gc-nrf-nrf-6c8cc8b69-q868p             1/1     Running    0          81m
pod/free5gc-helm-free5gc-nssf-nssf-5c9d76fc69-gmnqz          1/1     Running    0          81m
pod/free5gc-helm-free5gc-pcf-pcf-78f7dbc67d-96d9x            1/1     Running    0          81m
pod/free5gc-helm-free5gc-smf-smf-5dbcc8565c-pr9fj            1/1     Running    0          81m
pod/free5gc-helm-free5gc-udm-udm-68ff9fbd47-jldfk            1/1     Running    0          81m
pod/free5gc-helm-free5gc-udr-udr-5bd79d98f8-c9pv4            1/1     Running    0          81m
pod/free5gc-helm-free5gc-upf-upf1-58fbf87ff-7c8mf            1/1     Running    0          81m
pod/free5gc-helm-free5gc-upf-upf2-6bddf9969f-h5j57           1/1     Running    0          81m
pod/free5gc-helm-free5gc-upf-upfb-6b8cdb8fcc-89qp4           1/1     Running    0          81m
pod/free5gc-helm-free5gc-webui-webui-75c45c779c-d5hdq        1/1     Running    0          81m
pod/mongodb-0                                                1/1     Running    0          81m
pod/ueransim-1-gnb-64f6ddfc54-99bxg                          0/1     Init:0/1   0          21m
pod/ueransim-1-ue-59bdcb9599-8m7ll                           1/1     Running    0          31m

NAME                                        TYPE           CLUSTER-IP       EXTERNAL-IP         PORT(S)            AGE
service/free5gc-helm-free5gc-amf-amf-n2     LoadBalancer   10.152.183.63    llb-10.100.50.249   38412:31412/SCTP   81m
service/free5gc-helm-free5gc-amf-service    ClusterIP      10.152.183.246   <none>              80/TCP             81m
service/free5gc-helm-free5gc-amf2-amf2-n2   LoadBalancer   10.152.183.113   llb-10.100.50.249   38412:31413/SCTP   81m
service/free5gc-helm-free5gc-amf2-service   ClusterIP      10.152.183.144   <none>              80/TCP             81m
service/free5gc-helm-free5gc-ausf-service   ClusterIP      10.152.183.196   <none>              80/TCP             81m
service/free5gc-helm-free5gc-chf-service    ClusterIP      10.152.183.89    <none>              80/TCP             81m
service/free5gc-helm-free5gc-nef-service    ClusterIP      10.152.183.143   <none>              80/TCP             81m
service/free5gc-helm-free5gc-nssf-service   ClusterIP      10.152.183.73    <none>              80/TCP             81m
service/free5gc-helm-free5gc-pcf-service    ClusterIP      10.152.183.214   <none>              80/TCP             81m
service/free5gc-helm-free5gc-smf-service    ClusterIP      10.152.183.81    <none>              80/TCP             81m
service/free5gc-helm-free5gc-udm-service    ClusterIP      10.152.183.223   <none>              80/TCP             81m
service/free5gc-helm-free5gc-udr-service    ClusterIP      10.152.183.199   <none>              80/TCP             81m
service/gnb-service                         ClusterIP      10.152.183.59    <none>              4997/UDP           31m
service/mongodb                             ClusterIP      10.152.183.202   <none>              27017/TCP          81m
service/nrf-nnrf                            ClusterIP      10.152.183.209   <none>              8000/TCP           81m
service/webui-nbiling                       ClusterIP      10.152.183.242   <none>              2122/TCP           81m
service/webui-ncgf                          ClusterIP      10.152.183.104   <none>              2121/TCP           81m
service/webui-service                       NodePort       10.152.183.124   <none>              5000:30500/TCP     81m

NAME                                                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/free5gc-helm-free5gc-amf-amf             1/1     1            1           81m
deployment.apps/free5gc-helm-free5gc-amf2-amf2           1/1     1            1           81m
deployment.apps/free5gc-helm-free5gc-ausf-ausf           1/1     1            1           81m
deployment.apps/free5gc-helm-free5gc-chf-chf             1/1     1            1           81m
deployment.apps/free5gc-helm-free5gc-dbpython-dbpython   1/1     1            1           81m
deployment.apps/free5gc-helm-free5gc-nef-nef             1/1     1            1           81m
deployment.apps/free5gc-helm-free5gc-nrf-nrf             1/1     1            1           81m
deployment.apps/free5gc-helm-free5gc-nssf-nssf           1/1     1            1           81m
deployment.apps/free5gc-helm-free5gc-pcf-pcf             1/1     1            1           81m
deployment.apps/free5gc-helm-free5gc-smf-smf             1/1     1            1           81m
deployment.apps/free5gc-helm-free5gc-udm-udm             1/1     1            1           81m
deployment.apps/free5gc-helm-free5gc-udr-udr             1/1     1            1           81m
deployment.apps/free5gc-helm-free5gc-upf-upf1            1/1     1            1           81m
deployment.apps/free5gc-helm-free5gc-upf-upf2            1/1     1            1           81m
deployment.apps/free5gc-helm-free5gc-upf-upfb            1/1     1            1           81m
deployment.apps/free5gc-helm-free5gc-webui-webui         1/1     1            1           81m
deployment.apps/ueransim-1-gnb                           0/1     1            0           31m
deployment.apps/ueransim-1-ue                            1/1     1            1           31m

NAME                                                               DESIRED   CURRENT   READY   AGE
replicaset.apps/free5gc-helm-free5gc-amf-amf-66ddf68dc9            1         1         1       81m
replicaset.apps/free5gc-helm-free5gc-amf2-amf2-687bd6879b          1         1         1       81m
replicaset.apps/free5gc-helm-free5gc-ausf-ausf-64c684f546          1         1         1       81m
replicaset.apps/free5gc-helm-free5gc-chf-chf-7c7bb88fb7            1         1         1       81m
replicaset.apps/free5gc-helm-free5gc-dbpython-dbpython-59684d749   1         1         1       81m
replicaset.apps/free5gc-helm-free5gc-nef-nef-759b6dfbdb            1         1         1       81m
replicaset.apps/free5gc-helm-free5gc-nrf-nrf-6c8cc8b69             1         1         1       81m
replicaset.apps/free5gc-helm-free5gc-nssf-nssf-5c9d76fc69          1         1         1       81m
replicaset.apps/free5gc-helm-free5gc-pcf-pcf-78f7dbc67d            1         1         1       81m
replicaset.apps/free5gc-helm-free5gc-smf-smf-5dbcc8565c            1         1         1       81m
replicaset.apps/free5gc-helm-free5gc-udm-udm-68ff9fbd47            1         1         1       81m
replicaset.apps/free5gc-helm-free5gc-udr-udr-5bd79d98f8            1         1         1       81m
replicaset.apps/free5gc-helm-free5gc-upf-upf1-58fbf87ff            1         1         1       81m
replicaset.apps/free5gc-helm-free5gc-upf-upf2-6bddf9969f           1         1         1       81m
replicaset.apps/free5gc-helm-free5gc-upf-upfb-6b8cdb8fcc           1         1         1       81m
replicaset.apps/free5gc-helm-free5gc-webui-webui-75c45c779c        1         1         1       81m
replicaset.apps/ueransim-1-gnb-64f6ddfc54                          1         1         0       31m
replicaset.apps/ueransim-1-ue-59bdcb9599                           1         1         1       31m

NAME                       READY   AGE
statefulset.apps/mongodb   1/1     81m

problem

gnb can't find AMF

vboxuser@test1000:~/work/free5gc-helm/charts$ docker exec -it loxilb loxicmd get lb -o wide
|    EXT IP     | SEC IPS | HOST | PORT  | PROTO |                        NAME                         | MARK | SEL |   MODE    | ENDPOINT  | EPORT | WEIGHT | STATE | COUNTERS |
|---------------|---------|------|-------|-------|-----------------------------------------------------|------|-----|-----------|-----------|-------|--------|-------|----------|
| 10.100.50.249 |         |      | 38412 | sctp  | free5gc_free5gc-helm-free5gc-amf2-amf2-n2:llb-inst0 |    0 | n2  | fullproxy | 10.0.2.15 | 31413 |      1 | -     | 0:0      |


vboxuser@test1000:~/work/free5gc-helm/charts$ docker exec -it loxilb loxicmd get ct
| SERVICE NAME | DESTIP | SRCIP | DPORT | SPORT | PROTO | STATE | ACT | PACKETS | BYTES |
|--------------|--------|-------|-------|-------|-------|-------|-----|---------|-------|

two loadbalcencer

vboxuser@test1000:~/work/free5gc-helm/charts$ kubectl get services -n free5gc
NAME                                TYPE           CLUSTER-IP       EXTERNAL-IP         PORT(S)            AGE
free5gc-helm-free5gc-amf-amf-n2     LoadBalancer   10.152.183.63    llb-10.100.50.249   38412:31412/SCTP   93m
free5gc-helm-free5gc-amf-service    ClusterIP      10.152.183.246   <none>              80/TCP             93m
free5gc-helm-free5gc-amf2-amf2-n2   LoadBalancer   10.152.183.113   llb-10.100.50.249   38412:31413/SCTP   93m
free5gc-helm-free5gc-amf2-service   ClusterIP      10.152.183.144   <none>              80/TCP             93m

maybe i should just use original amf service instead of loadbalancer type

  amf:
    n2if:  # NGAP
      ipAddress: 10.100.50.250
    service:
      ngap:
        enabled: false
        name: amf-n2
        port: 38412
        nodeport: 31412
        protocol: SCTP
        type: NodePort

create lb by my own and still can't establish connection

docker exec -it loxilb loxicmd create lb 10.100.50.249 --sctp=38412:38412 --endpoints=10.100.50.251:1,10.100.50.252:1 --mode=fullproxy

vboxuser@test1000:~/work/free5gc-helm$ docker exec -it loxilb loxicmd create lb 10.100.50.249 --sctp=38412:38412 --endpoints=10.100.50.251:1,10.100.50.252:1 --mode=fullproxy

ProtoPortpair: map[sctp:[38412:38412]]
Debug: response.StatusCode: 200
Success
vboxuser@test1000:~/work/free5gc-helm$ docker exec -it loxilb loxicmd get lb -o wide
|    EXT IP     | SEC IPS | HOST | PORT  | PROTO |                        NAME                         | MARK | SEL |   MODE    |   ENDPOINT    | EPORT | WEIGHT | STATE | COUNTERS |
|---------------|---------|------|-------|-------|-----------------------------------------------------|------|-----|-----------|---------------|-------|--------|-------|----------|
| 10.100.50.249 |         |      | 38412 | sctp  | free5gc_free5gc-helm-free5gc-amf2-amf2-n2:llb-inst0 |    0 | rr  | fullproxy | 10.100.50.251 | 38412 |      1 | -     | 0:0      |
|               |         |      |       |       |                                                     |      |     |           | 10.100.50.252 | 38412 |      1 | -     | 0:0      |
vboxuser@test1000:~/work/free5gc-helm$

Questions:

  1. Is my LoxiLB configuration correct for SCTP load balancing?
  2. Why isn't the SCTP connection being established through the VIP?
  3. I think I might be mixing up in-cluster and external-cluster LoxiLB deployment patterns
  4. the way I duplicate amf may wrong? like use NodePort instead of LoadBalancer type for AMF services?

Any help or guidance would be greatly appreciated!! or if need more info.

Last changed by 
 
william linn
major in CS
0
244

Read more

Hands-On with sched_ext: Building Custom eBPF CPU Schedulers

In part 1: Exploring sched_ext: BPF-Powered CPU Schedulers in the Linux Kernel, we explored the architecture and concepts behind sched_ext, Linux's framework for implementing custom CPU schedulers using eBPF. We examined how this revolutionary approach allows dynamically loading schedulers without kernel recompilation, and we compared different implementation styles including C-based (scx_simple), Rust-based (scx_bpfland,scx_rustland), and even potential Go implementations SCX GoLand Core

May 9, 2025
Exploring sched_ext: BPF-Powered CPU Schedulers in the Linux Kernel

In today's complex computing environments, the default Linux scheduler doesn't always provide optimal performance for specialized workloads. This is where sched_ext (SCX) comes in – a framework that allows implementing custom CPU schedulers in BPF (Berkeley Packet Filter) and loading them dynamically. In this technical analysis.

Mar 8, 2025
NGAP Load Balancing with LoxiLB and free5GC

Single Kubernetes Cluster: All Free5GC network functions run in a single K8s cluster

Feb 16, 2025
loxilb ebpf

uses eBPF to dynamically modify the packet processing rules based on the current network conditions and traffic patterns.

Nov 30, 2024
Read more from william linn
Published on HackMD