# NGAP Load Balancing with LoxiLB and free5GC (L7 load-balancer)
Based on [NGAP Load Balancing with LoxiLB](https://www.loxilb.io/post/ngap-load-balancing-with-loxilb) and [5g-sctp-loadbalancer](https://medium.com/@ben0978327139/5g-sctp-loadbalancer-using-loxilb-applying-on-free5gc-b5c05bb723f0), but change 5G Core to use **free5gc** instead of Open5gs

This implementation is primarily based on above two references

## setup and target:

I'm trying to implement SCTP load balancing for Free5GC AMF using LoxiLB, but encountering connection issues. Here's my setup:

Environment:
- Single Kubernetes cluster running Free5GC components
- LoxiLB running as standalone Docker container
- Two AMF instances
- UERANSIM for testing

Configuration:
1. LoxiLB VIP: 10.100.50.249
2. AMF1: 10.100.50.251
3. AMF2: 10.100.50.252

# Overall Architecture (Single-Cluster)

* Single Kubernetes Cluster: All Free5GC network functions run in a single K8s cluster
* LoxiLB: LoxiLB will within that same cluster to load-balance traffic destined for the AMF’s N2 interface (SCTP).

# Preparing docker LoxiLB instance
LoxiLB runs as a standalone Docker container.

docker run -u root --cap-add SYS_ADMIN --restart unless-stopped --privileged -dit -v /dev/log:/dev/log --entrypoint=/root/loxilb-io/loxilb/loxilb --net=host --name loxilb ghcr.io/loxilb-io/loxilb:scp --proxyonlymode

# Deploy kube-loxilb
1. **loxiURL**: the IP address of the machine running the LoxiLB data-plane and to talk to the external LoxiLB instance.
`Host network IP: 192.168.56.106 (enp0s8)`

        args:
        - --loxiURL=http://192.168.56.106:11111
        - --cidrPools=defaultPool=10.100.50.249/32
        #- --monitor
        #- --setBGP=64512
        #- --extBGPPeers=50.50.50.1:65101,51.51.51.1:65102
        #- --setRoles
        - --setLBMode=2
        #- --config=/opt/loxilb/agent/kube-loxilb.conf

kubectl apply -f kube-loxilb.yaml

## verify status

vboxuser@test1000:~/work/free5gc-helm/charts$ docker ps
CONTAINER ID   IMAGE                          COMMAND                  CREATED        STATUS             PORTS     NAMES
5b1a2ef6d7e0   ghcr.io/loxilb-io/loxilb:scp   "/root/loxilb-io/lox…"   41 hours ago   Up About an hour             loxilb
vboxuser@test1000:~/work/free5gc-helm/charts$ kubectl get pod -n kube-system
NAME                                     READY   STATUS    RESTARTS       AGE
calico-kube-controllers-77bd7c5b-2fmzs   1/1     Running   30 (25h ago)   27d
calico-node-jhvc4                        1/1     Running   12 (25h ago)   5d11h
coredns-5cf7b49c88-5vwcz                 1/1     Running   10 (25h ago)   5d5h
kube-loxilb-7bb475df98-s8zrb             1/1     Running   0              38m
kube-multus-ds-5j2k4                     1/1     Running   22 (25h ago)   27d

# Deploy free5gC and ueransim using helm

## Modify the AMF service
Modify the AMF service to use LoadBalancer type and add LoxiLB annotations

`free5gc/values.yaml`

change:

  amf:
    n2if:  # NGAP
      ipAddress: 10.100.50.250
    service:
      ngap:
        enabled: false
        name: amf-n2
        port: 38412
        nodeport: 31412
        protocol: SCTP
        type: NodePort

to:

  amf:
    n2if:  # NGAP
      ipAddress: 10.100.50.251
    service:
      ngap:
        enabled: true
        name: amf-n2
        port: 38412
        protocol: SCTP
        type: LoadBalancer
        loadBalancerClass: "loxilb.io/loxilb"  # 
        annotations:
          loxilb.io/probetype: "none"
          loxilb.io/lbmode: "fullproxy"
          loxilb.io/epselect: "n2"
          loxilb.io/lbclass: "loxilb.io/loxilb" 
  amf2:
    n2if:  # NGAP
      ipAddress: 10.100.50.252
    service:
      ngap:
        enabled: true
        name: amf2-n2
        port: 38412
        protocol: SCTP
        type: LoadBalancer
        loadBalancerClass: "loxilb.io/loxilb"  # 
        annotations:
          loxilb.io/probetype: "none"
          loxilb.io/lbmode: "fullproxy"
          loxilb.io/epselect: "n2"
          loxilb.io/lbclass: "loxilb.io/loxilb" 

### create amf2 folder to apply amf2
then run:

helm install -n free5gc free5gc-helm ./free5gc/ \
--set global.n6network.masterIf=enp0s8 \
--set global.n6network.subnetIP="192.168.56.0" \
--set global.n6network.gatewayIP="192.168.56.1" \
--set free5gc-upf.upf1.n6if.ipAddress="192.168.56.200" \
--set free5gc-upf.upf2.n6if.ipAddress="192.168.56.201" \
--set free5gc-upf.upfb.n6if.ipAddress="192.168.56.202" \
--set global.n2network.masterIf=enp0s8 \
--set global.n3network.masterIf=enp0s8 \
--set global.n4network.masterIf=enp0s8 \
--set global.n9network.masterIf=enp0s8

## verify the free5gc and ueransim
see many problem

vboxuser@test1000:~/work/free5gc-helm/charts$ kubectl get all -n free5gc 
NAME                                                         READY   STATUS     RESTARTS   AGE
pod/free5gc-helm-free5gc-amf-amf-66ddf68dc9-rjtht            1/1     Running    0          81m
pod/free5gc-helm-free5gc-amf2-amf2-687bd6879b-zdj68          1/1     Running    0          81m
pod/free5gc-helm-free5gc-ausf-ausf-64c684f546-5psjf          1/1     Running    0          81m
pod/free5gc-helm-free5gc-chf-chf-7c7bb88fb7-89lwh            1/1     Running    0          81m
pod/free5gc-helm-free5gc-dbpython-dbpython-59684d749-6pkv6   1/1     Running    0          81m
pod/free5gc-helm-free5gc-nef-nef-759b6dfbdb-w5fwm            1/1     Running    0          81m
pod/free5gc-helm-free5gc-nrf-nrf-6c8cc8b69-q868p             1/1     Running    0          81m
pod/free5gc-helm-free5gc-nssf-nssf-5c9d76fc69-gmnqz          1/1     Running    0          81m
pod/free5gc-helm-free5gc-pcf-pcf-78f7dbc67d-96d9x            1/1     Running    0          81m
pod/free5gc-helm-free5gc-smf-smf-5dbcc8565c-pr9fj            1/1     Running    0          81m
pod/free5gc-helm-free5gc-udm-udm-68ff9fbd47-jldfk            1/1     Running    0          81m
pod/free5gc-helm-free5gc-udr-udr-5bd79d98f8-c9pv4            1/1     Running    0          81m
pod/free5gc-helm-free5gc-upf-upf1-58fbf87ff-7c8mf            1/1     Running    0          81m
pod/free5gc-helm-free5gc-upf-upf2-6bddf9969f-h5j57           1/1     Running    0          81m
pod/free5gc-helm-free5gc-upf-upfb-6b8cdb8fcc-89qp4           1/1     Running    0          81m
pod/free5gc-helm-free5gc-webui-webui-75c45c779c-d5hdq        1/1     Running    0          81m
pod/mongodb-0                                                1/1     Running    0          81m
pod/ueransim-1-gnb-64f6ddfc54-99bxg                          0/1     Init:0/1   0          21m
pod/ueransim-1-ue-59bdcb9599-8m7ll                           1/1     Running    0          31m

NAME                                        TYPE           CLUSTER-IP       EXTERNAL-IP         PORT(S)            AGE
service/free5gc-helm-free5gc-amf-amf-n2     LoadBalancer   10.152.183.63    llb-10.100.50.249   38412:31412/SCTP   81m
service/free5gc-helm-free5gc-amf-service    ClusterIP      10.152.183.246   <none>              80/TCP             81m
service/free5gc-helm-free5gc-amf2-amf2-n2   LoadBalancer   10.152.183.113   llb-10.100.50.249   38412:31413/SCTP   81m
service/free5gc-helm-free5gc-amf2-service   ClusterIP      10.152.183.144   <none>              80/TCP             81m
service/free5gc-helm-free5gc-ausf-service   ClusterIP      10.152.183.196   <none>              80/TCP             81m
service/free5gc-helm-free5gc-chf-service    ClusterIP      10.152.183.89    <none>              80/TCP             81m
service/free5gc-helm-free5gc-nef-service    ClusterIP      10.152.183.143   <none>              80/TCP             81m
service/free5gc-helm-free5gc-nssf-service   ClusterIP      10.152.183.73    <none>              80/TCP             81m
service/free5gc-helm-free5gc-pcf-service    ClusterIP      10.152.183.214   <none>              80/TCP             81m
service/free5gc-helm-free5gc-smf-service    ClusterIP      10.152.183.81    <none>              80/TCP             81m
service/free5gc-helm-free5gc-udm-service    ClusterIP      10.152.183.223   <none>              80/TCP             81m
service/free5gc-helm-free5gc-udr-service    ClusterIP      10.152.183.199   <none>              80/TCP             81m
service/gnb-service                         ClusterIP      10.152.183.59    <none>              4997/UDP           31m
service/mongodb                             ClusterIP      10.152.183.202   <none>              27017/TCP          81m
service/nrf-nnrf                            ClusterIP      10.152.183.209   <none>              8000/TCP           81m
service/webui-nbiling                       ClusterIP      10.152.183.242   <none>              2122/TCP           81m
service/webui-ncgf                          ClusterIP      10.152.183.104   <none>              2121/TCP           81m
service/webui-service                       NodePort       10.152.183.124   <none>              5000:30500/TCP     81m

NAME                                                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/free5gc-helm-free5gc-amf-amf             1/1     1            1           81m
deployment.apps/free5gc-helm-free5gc-amf2-amf2           1/1     1            1           81m
deployment.apps/free5gc-helm-free5gc-ausf-ausf           1/1     1            1           81m
deployment.apps/free5gc-helm-free5gc-chf-chf             1/1     1            1           81m
deployment.apps/free5gc-helm-free5gc-dbpython-dbpython   1/1     1            1           81m
deployment.apps/free5gc-helm-free5gc-nef-nef             1/1     1            1           81m
deployment.apps/free5gc-helm-free5gc-nrf-nrf             1/1     1            1           81m
deployment.apps/free5gc-helm-free5gc-nssf-nssf           1/1     1            1           81m
deployment.apps/free5gc-helm-free5gc-pcf-pcf             1/1     1            1           81m
deployment.apps/free5gc-helm-free5gc-smf-smf             1/1     1            1           81m
deployment.apps/free5gc-helm-free5gc-udm-udm             1/1     1            1           81m
deployment.apps/free5gc-helm-free5gc-udr-udr             1/1     1            1           81m
deployment.apps/free5gc-helm-free5gc-upf-upf1            1/1     1            1           81m
deployment.apps/free5gc-helm-free5gc-upf-upf2            1/1     1            1           81m
deployment.apps/free5gc-helm-free5gc-upf-upfb            1/1     1            1           81m
deployment.apps/free5gc-helm-free5gc-webui-webui         1/1     1            1           81m
deployment.apps/ueransim-1-gnb                           0/1     1            0           31m
deployment.apps/ueransim-1-ue                            1/1     1            1           31m

NAME                                                               DESIRED   CURRENT   READY   AGE
replicaset.apps/free5gc-helm-free5gc-amf-amf-66ddf68dc9            1         1         1       81m
replicaset.apps/free5gc-helm-free5gc-amf2-amf2-687bd6879b          1         1         1       81m
replicaset.apps/free5gc-helm-free5gc-ausf-ausf-64c684f546          1         1         1       81m
replicaset.apps/free5gc-helm-free5gc-chf-chf-7c7bb88fb7            1         1         1       81m
replicaset.apps/free5gc-helm-free5gc-dbpython-dbpython-59684d749   1         1         1       81m
replicaset.apps/free5gc-helm-free5gc-nef-nef-759b6dfbdb            1         1         1       81m
replicaset.apps/free5gc-helm-free5gc-nrf-nrf-6c8cc8b69             1         1         1       81m
replicaset.apps/free5gc-helm-free5gc-nssf-nssf-5c9d76fc69          1         1         1       81m
replicaset.apps/free5gc-helm-free5gc-pcf-pcf-78f7dbc67d            1         1         1       81m
replicaset.apps/free5gc-helm-free5gc-smf-smf-5dbcc8565c            1         1         1       81m
replicaset.apps/free5gc-helm-free5gc-udm-udm-68ff9fbd47            1         1         1       81m
replicaset.apps/free5gc-helm-free5gc-udr-udr-5bd79d98f8            1         1         1       81m
replicaset.apps/free5gc-helm-free5gc-upf-upf1-58fbf87ff            1         1         1       81m
replicaset.apps/free5gc-helm-free5gc-upf-upf2-6bddf9969f           1         1         1       81m
replicaset.apps/free5gc-helm-free5gc-upf-upfb-6b8cdb8fcc           1         1         1       81m
replicaset.apps/free5gc-helm-free5gc-webui-webui-75c45c779c        1         1         1       81m
replicaset.apps/ueransim-1-gnb-64f6ddfc54                          1         1         0       31m
replicaset.apps/ueransim-1-ue-59bdcb9599                           1         1         1       31m

NAME                       READY   AGE
statefulset.apps/mongodb   1/1     81m

# problem
## gnb can't find AMF

vboxuser@test1000:~/work/free5gc-helm/charts$ docker exec -it loxilb loxicmd get lb -o wide
|    EXT IP     | SEC IPS | HOST | PORT  | PROTO |                        NAME                         | MARK | SEL |   MODE    | ENDPOINT  | EPORT | WEIGHT | STATE | COUNTERS |
|---------------|---------|------|-------|-------|-----------------------------------------------------|------|-----|-----------|-----------|-------|--------|-------|----------|
| 10.100.50.249 |         |      | 38412 | sctp  | free5gc_free5gc-helm-free5gc-amf2-amf2-n2:llb-inst0 |    0 | n2  | fullproxy | 10.0.2.15 | 31413 |      1 | -     | 0:0      |


vboxuser@test1000:~/work/free5gc-helm/charts$ docker exec -it loxilb loxicmd get ct
| SERVICE NAME | DESTIP | SRCIP | DPORT | SPORT | PROTO | STATE | ACT | PACKETS | BYTES |
|--------------|--------|-------|-------|-------|-------|-------|-----|---------|-------|

## two loadbalcencer

vboxuser@test1000:~/work/free5gc-helm/charts$ kubectl get services -n free5gc
NAME                                TYPE           CLUSTER-IP       EXTERNAL-IP         PORT(S)            AGE
free5gc-helm-free5gc-amf-amf-n2     LoadBalancer   10.152.183.63    llb-10.100.50.249   38412:31412/SCTP   93m
free5gc-helm-free5gc-amf-service    ClusterIP      10.152.183.246   <none>              80/TCP             93m
free5gc-helm-free5gc-amf2-amf2-n2   LoadBalancer   10.152.183.113   llb-10.100.50.249   38412:31413/SCTP   93m
free5gc-helm-free5gc-amf2-service   ClusterIP      10.152.183.144   <none>              80/TCP             93m

maybe i should just use original amf service instead of loadbalancer type

  amf:
    n2if:  # NGAP
      ipAddress: 10.100.50.250
    service:
      ngap:
        enabled: false
        name: amf-n2
        port: 38412
        nodeport: 31412
        protocol: SCTP
        type: NodePort

## create lb by my own and still can't establish connection

docker exec -it loxilb loxicmd create lb 10.100.50.249 --sctp=38412:38412 --endpoints=10.100.50.251:1,10.100.50.252:1 --mode=fullproxy

vboxuser@test1000:~/work/free5gc-helm$ docker exec -it loxilb loxicmd create lb 10.100.50.249 --sctp=38412:38412 --endpoints=10.100.50.251:1,10.100.50.252:1 --mode=fullproxy

ProtoPortpair: map[sctp:[38412:38412]]
Debug: response.StatusCode: 200
Success
vboxuser@test1000:~/work/free5gc-helm$ docker exec -it loxilb loxicmd get lb -o wide
|    EXT IP     | SEC IPS | HOST | PORT  | PROTO |                        NAME                         | MARK | SEL |   MODE    |   ENDPOINT    | EPORT | WEIGHT | STATE | COUNTERS |
|---------------|---------|------|-------|-------|-----------------------------------------------------|------|-----|-----------|---------------|-------|--------|-------|----------|
| 10.100.50.249 |         |      | 38412 | sctp  | free5gc_free5gc-helm-free5gc-amf2-amf2-n2:llb-inst0 |    0 | rr  | fullproxy | 10.100.50.251 | 38412 |      1 | -     | 0:0      |
|               |         |      |       |       |                                                     |      |     |           | 10.100.50.252 | 38412 |      1 | -     | 0:0      |
vboxuser@test1000:~/work/free5gc-helm$ 

### Questions:

1. Is my LoxiLB configuration correct for SCTP load balancing?
1. Why isn't the SCTP connection being established through the VIP?
1. I think I might be mixing up in-cluster and external-cluster LoxiLB deployment patterns
1. the way I duplicate amf may wrong? like use NodePort instead of LoadBalancer type for AMF services?

**Any help or guidance would be greatly appreciated!! or if need more info.**