As a VNF 5G Implementation Division on Kubernetes, I design and deploy high-performance OpenAirInterface (OAI) 5G use cases. I optimize network performance, ensure security integration, and collaborate on defining use cases. My focus is on automation for efficient seamless deployment 5G integration in Kubernetes environments.
Open Netra, standing for Open Network Training, is a comprehensive solution designed to facilitate the training, simulation, and management of 5G networks. The primary functionalities of Open Netra include E2E (End-to-End) 5G dashboard simulation and RAN (Radio Access Network) monitoring & configuration. The entire infrastructure is built on a foundation of virtualized network functions, specifically implemented as OpenAirInterface containers, orchestrated and managed by Kubernetes.
Production Ready Kubernetes Cluster
(Optional) Multus CNI if using multiple interfaces for NFs
Kube-Sniff & Wireshark
Setup Krew (kubectl package-manager)
Make sure git are installed
sudo apt install git
Run this command to download and install krew
(
set -x; cd "$(mktemp -d)" &&
OS="$(uname | tr '[:upper:]' '[:lower:]')" &&
ARCH="$(uname -m | sed -e 's/x86_64/amd64/' -e 's/\(arm\)\(64\)\?.*/\1\2/' -e 's/aarch64$/arm64/')" &&
KREW="krew-${OS}_${ARCH}" &&
curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/${KREW}.tar.gz" &&
tar zxvf "${KREW}.tar.gz" &&
./"${KREW}" install krew
)
Add the $HOME/.krew/bin directory to your PATH environment variable.
export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"
Install Kubectl-Sniff
Install it from krew package manager
kubectl krew install sniff
Install Wireshark
Setup wireshark from os repository
sudo apt install wireshark -y
Configure the permission problem
sudo gpasswd -a $USER wireshark
sudo adduser $USER wireshark
sudo setcap 'CAP_NET_RAW+eip CAP_NET_ADMIN+eip' /usr/bin/dumpcap
How to use ksniff
If used in Wireshark GUI
kubectl sniff oai-nr-ue-2-786b494559-jtz97 -n oai
If used in Terminal
kubectl sniff oai-nr-ue-2-786b494559-jtz97 -n oai -f "tcp" -o - | tshark -r -
INFO[0000] using tcpdump path at: '/home/rafli/.krew/store/sniff/v1.6.2/static-tcpdump'
INFO[0000] no container specified, taking first container we found in pod.
INFO[0000] selected container: 'nr-ue'
INFO[0000] sniffing method: upload static tcpdump
INFO[0000] sniffing on pod: 'oai-nr-ue-2-786b494559-jtz97' [namespace: 'oai', container: 'nr-ue', filter: 'tcp', interface: 'any']
INFO[0000] uploading static tcpdump binary from: '/home/rafli/.krew/store/sniff/v1.6.2/static-tcpdump' to: '/tmp/static-tcpdump'
INFO[0000] uploading file: '/home/rafli/.krew/store/sniff/v1.6.2/static-tcpdump' to '/tmp/static-tcpdump' on container: 'nr-ue'
INFO[0000] executing command: '[/bin/sh -c test -f /tmp/static-tcpdump]' on container: 'nr-ue', pod: 'oai-nr-ue-2-786b494559-jtz97', namespace: 'oai'
INFO[0000] command: '[/bin/sh -c test -f /tmp/static-tcpdump]' executing successfully exitCode: '0', stdErr :''
INFO[0000] file found: ''
INFO[0000] file was already found on remote pod
INFO[0000] tcpdump uploaded successfully
INFO[0000] output file option specified, storing output in: '-'
INFO[0000] start sniffing on remote container
INFO[0000] executing command: '[/tmp/static-tcpdump -i any -U -w - tcp]' on container: 'nr-ue', pod: 'oai-nr-ue-2-786b494559-jtz97', namespace: 'oai'
1 0.000000 10.233.75.62 → 10.233.75.54 TCP 100 44242 → 4043 [PSH, ACK] Seq=1 Ack=1 Win=24091 Len=32 TSval=2131950554 TSecr=1253996121
2 0.000066 10.233.75.62 → 10.233.75.54 TCP 64376 44242 → 4043 [PSH, ACK] Seq=33 Ack=1 Win=24091 Len=64308 TSval=2131950554 TSecr=1253996121
3 0.000100 10.233.75.54 → 10.233.75.62 TCP 68 4043 → 44242 [ACK] Seq=1 Ack=64341 Win=16287 Len=0 TSval=1253996122 TSecr=2131950554
Deployment Plan
Core Multi Interface
Purpose | Interface | IP Address | Node |
---|---|---|---|
AMF - N2 | ens33 | 172.21.6.94/22 | 3 |
UPF - N3 | ens33 | 172.21.8.95/22 | 3 |
UPF - N4 | ens33 | 192.168.24.2/24 | 3 |
UPF - N6 | ens33 | 192.168.22.2/24 | 3 |
SMF - N6 | ens33 | 192.168.24.3/24 | 3 |
Helm Step Installation
I installed the basic core network setup. Because it support to utilize external database for network function.
Clone Core 5G Repository
git clone https://github.com/adaptivenetworklab/AN-OPEN-NETRA-VNF
Create Namespace named core-network
kubectl create ns core-network
Helm Custom Chart For Core Network Deployment
# Core Network Chart
cd AN-OPEN-NETRA-VNF/oai-5g-core/
mysql oai-5g-advance oai-5g-basic oai-5g-mini oai-amf oai-ausf oai-nrf oai-nssf oai-smf oai-traffic-server oai-udm oai-udr oai-upf
helm dependency update
# RAN Chart
ls AN-OPEN-NETRA-VNF/user_n/oai-e2e/
oai-cu oai-du oai-nr-ue
helm dependency update
Starting version 2.0.0
of OAI 5G Core network functions their configuration will be in config.yaml
and all infrastructure related information including image definition will be in values.yaml
.
Helm chart of every network function looks similar and has the below structure. Only the chart of mysql database and NRF is different.
Configuring Core Network
Network function discovers each-other using NRF and instead of using the ip-address of network functions we rely on using their FQDN, Kubernetes service concept. To communicate with each other whether we deploy them in reference point architecture or service based architecture.
For example: AMF registers with NRF using NRF FQDN (oai-nrf.core-network.svc.cluster.local
). This way we can get rid of any static ip-address configuration.
Configure Multiple Interfaces
The network functions will use different virtual ethernet (veth) interfaces to bind their different logical interface. Example AMF communicates with gNB using N2 and with SMF and NRF using Namf, the Service Base Interface (SBI).
This type of configuration is also used when gNB is outside of the cluster or UPF is outside of the cluster.
To make the above seperation we are using multus to provide multiple ethernet interfaces to network functions which have multiple communication interfaces.
Only AMF, SMF and UPF have the possiblity to use multus. Other network functions can also use multus but then it needs to be configured.
To configure multus for AMF, SMF or UPF, in values.yaml
of each network function edit the multus section.
## Example from oai-5g-basic/values.yaml
multus:
## If you don't want to add a default route in your pod then leave this field empty
defaultGateway: ""
n2Interface:
create: true
Ipadd: "172.21.6.94"
Netmask: "22"
## If you do not have a gateway leave the field empty
Gateway:
## If you do not want to add any routes in your pod then leave this field empty
routes:
hostInterface: "ens33" # Interface of the host machine on which this pod will be scheduled
Configure Helm Chart Parameters
In the config.yaml of oai-5g-basic helm charts you will see the configurable parameters for all the network functions check, the PLMN, DNN and subscriber information in mysql database
For basic and advance deployment check the database oai_db-basic.sql
A new subscriber entry can be added directly in the sql file or it can be added once the core network is already deployed.
To add the entry before deploying the core network, make sure you have all the required subscriber information IMSI(ueid/supi), Key(encPermanentKey), OPC(encOpcKey), PLMN, NSSAI(SST, SD), DNN
vim AN-OPEN-NETRA-VNF/oai-5g-core/mysql/initialization/oai_db-basic.sql
# Add a new or edit existing entries after AuthenticationSubscription table
INSERT INTO `AuthenticationSubscription` (`ueid`, `authenticationMethod`, `encPermanentKey`, `protectionParameterId`, `sequenceNumber`, `authenticationManagementField`, `algorithmId`, `encOpcKey`, `encTopcKey`, `vectorGenerationInHss`, `n5gcAuthMethod`, `rgAuthenticationInd`, `supi`) VALUES
('208990100001101', '5G_AKA', 'a92d71ed92f43ebae1720f34fd2e6966', 'a92d71ed92f43ebae1720f34fd2e6966', '{\"sqn\": \"000000000020\", \"sqnScheme\": \"NON_TIME_BASED\", \"lastIndexes\": {\"ausf\": 0}}', '8000', 'milenage', 'C42449363BBAD02B66D16BC975D77CC1', NULL, NULL, NULL, NULL, '208990100001101'),
# Add the PDN/DNN information after SessionManagementSubscriptionData table
# To assign a static ip-address use the below entry
INSERT INTO `SessionManagementSubscriptionData` (`ueid`, `servingPlmnid`, `singleNssai`, `dnnConfigurations`, `internalGroupIds`, `sharedVnGroupDataIds`, `sharedDnnConfigurationsId`, `odbPacketServices`, `traceData`, `sharedTraceDataId`, `expectedUeBehavioursList`, `suggestedPacketNumDlList`, `3gppChargingCharacteristics`) VALUES
('208990100001101', '20899', '{\"sd\": \"16777204\", \"sst\": 1}', '{\"ims\": {\"sscModes\": {\"defaultSscMode\": \"SSC_MODE_1\"}, \"sessionAmbr\": {\"uplink\": \"1000Mbps\", \"downlink\": \"1000Mbps\"}, \"5gQosProfile\": {\"5qi\": 2, \"arp\": {\"preemptCap\": \"NOT_PREEMPT\", \"preemptVuln\": \"PREEMPTABLE\", \"priorityLevel\": 15}, \"priorityLevel\": 1}, \"pduSessionTypes\": {\"defaultSessionType\": \"IPV4V6\"}}, \"oai\": {\"sscModes\": {\"defaultSscMode\": \"SSC_MODE_1\"}, \"sessionAmbr\": {\"uplink\": \"1000Mbps\", \"downlink\": \"1000Mbps\"}, \"5gQosProfile\": {\"5qi\": 1, \"arp\": {\"preemptCap\": \"NOT_PREEMPT\", \"preemptVuln\": \"PREEMPTABLE\", \"priorityLevel\": 15}, \"priorityLevel\": 1}, \"pduSessionTypes\": {\"defaultSessionType\": \"IPV4\"}, \"staticIpAddress\": [{\"ipv4Addr\": \"12.1.1.100\"}]}}', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL);
In the config file smf.use_local_subscription_info
should be yes
to use the user DNN subscription information from the database. Else it will be used as defined in the configuration file.
Helm install with related chart
Helm charts have an order of deployment for the proper configuration of core network.
Once the configuration is finished the charts can be deployed with a user who has the rights to
Install with this helm command:
cd AN-OPEN-NETRA-VNF/oai-5g-core/
helm install basic oai-5g-basic/ --namespace core-network
NAME: basic
LAST DEPLOYED: Tue Dec 12 10:04:40 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
This command can take around 3-5 mins depending on your network speed and cluster configuration (computational capacity). You can use the wait command to see if the core network functions are running or not
kubectl wait --for=condition=ready pod -l app.kubernetes.io/instance=basic --timeout=3m -n core-network
Check the logs smf
and upf
to see that the PFCP session is properly configured,
kubectl logs -l app.kubernetes.io/name=oai-smf -n core-network | grep 'handle_receive(16 bytes)' | wc -l
kubectl logs -l app.kubernetes.io/name=oai-upf -n core-network | grep 'handle_receive(16 bytes)' | wc -l
If the value is more than 1
for both then it will verify that smf
and upf
have successfully registered to nrf
and there is a PFCP session.
Wait until all the pods are up and running (k9s)
Configuring PhpMyAdmin for Subscriber Management
Deploy the secret first
kubectl apply -f phypmyadmin/secret.yaml
Deploy the rest of the manifest
kubectl apply -f phypmyadmin/
PMA Dashboard
Setup your custom credential in the core manifest configuration file.
Deployment Plan: F1 Split RAN
CU Interface
Purpose | Interface | IP Address | Port |
---|---|---|---|
N1 & N2 | ens33 | 172.21.6.90/22 | 38472 |
N3 | ens33 | 172.21.8.91/22 | 2152 |
F1 | ens33 | 192.168.1.5/27 | –- |
DU & RU Interface
Purpose | Interface | IP Address | Port |
---|---|---|---|
F1 | ens33 | 192.168.1.6/27 | 38472 |
UE Interface
Purpose | Interface | IP Address | Port |
---|---|---|---|
Net1 | ens33 | 192.168.1.7/24 | 38472 |
Configmap CU File Templete
cu.conf: |
Active_gNBs = ( "{{ .Values.config.cuName}}");
# Asn1_verbosity, choice in: none, info, annoying
Asn1_verbosity = "none";
Num_Threads_PUSCH = 8;
gNBs =
(
{
////////// Identification parameters:
gNB_ID = 0xe04; ////////// Kalau bang nino: 0xe01
# cell_type = "CELL_MACRO_GNB";
gNB_name = "{{ .Values.config.cuName}}";
// Tracking area code, 0x0000 and 0xfffe are reserved values
tracking_area_code = {{ .Values.config.tac}};
plmn_list = ({ mcc = {{ .Values.config.mcc}}; mnc = {{ .Values.config.mnc}}; mnc_length = 2; snssaiList = ({ sst = {{ .Values.config.sst}}; }) });
nr_cellid = 12345678L;
tr_s_preference = "f1";
local_s_if_name = "{{ .Values.config.f1IfName}}";
local_s_address = "{{ .Values.multus.f1Interface.IPadd}}";
remote_s_address = "127.0.0.1";
local_s_portc = 501;
local_s_portd = {{ .Values.config.f1cuPort}};
remote_s_portc = 500;
remote_s_portd = {{ .Values.config.f1duPort}};
# ------- SCTP definitions
SCTP :
{
# Number of streams to use in input/output
SCTP_INSTREAMS = 2;
SCTP_OUTSTREAMS = 2;
};
////////// AMF parameters:
amf_ip_address = ( { ipv4 = "{{ .Values.config.amfhost}}";
ipv6 = "192:168:30::17";
active = "yes";
preference = "ipv4";
}
);
NETWORK_INTERFACES :
{
GNB_INTERFACE_NAME_FOR_NG_AMF = "{{ .Values.config.n2IfName}}";
GNB_IPV4_ADDRESS_FOR_NG_AMF = "{{ .Values.multus.n2Interface.IPadd}}";
GNB_INTERFACE_NAME_FOR_NGU = "{{ .Values.config.n3IfName}}";
GNB_IPV4_ADDRESS_FOR_NGU = "{{ .Values.multus.n3Interface.IPadd}}";
GNB_PORT_FOR_S1U = 2152; # Spec 2152
};
}
);
...
Conf DU File
du.conf: |
Active_gNBs = ( "{{ .Values.config.duName}}");
# Asn1_verbosity, choice in: none, info, annoying
Asn1_verbosity = "none";
gNBs =
(
{
////////// Identification parameters:
gNB_ID = 0xe05; ////////// Yang bang Nino: 0xe02
gNB_DU_ID = 0xe06;
# cell_type = "CELL_MACRO_GNB";
gNB_name = "{{ .Values.config.duName}}";
// Tracking area code, 0x0000 and 0xfffe are reserved values
tracking_area_code = {{ .Values.config.tac}} ;
plmn_list = ({ mcc = {{ .Values.config.mcc}}; mnc = {{ .Values.config.mnc}}; mnc_length = 2; snssaiList = ({ sst = {{ .Values.config.sst}}; }) });
nr_cellid = 12345678L;
////////// Physical parameters:
...
...
...
////////// Physical parameters:
# ------- SCTP definitions
SCTP :
{
# Number of streams to use in input/output
SCTP_INSTREAMS = 2;
SCTP_OUTSTREAMS = 2;
};
}
);
MACRLCs = (
{
num_cc = 1;
tr_s_preference = "local_L1";
tr_n_preference = "f1";
local_n_if_name = "{{ .Values.config.f1IfName}}";
local_n_address = "{{ .Values.multus.f1Interface.IPadd}}";
remote_n_address = "{{ .Values.config.cuHost}}";
local_n_portc = 500;
local_n_portd = {{ .Values.config.f1duPort}};
remote_n_portc = 501;
remote_n_portd = {{ .Values.config.f1cuPort}};
pusch_TargetSNRx10 = 200;
pucch_TargetSNRx10 = 200;
}
);
L1s = (
{
num_cc = 1;
tr_n_preference = "local_mac";
prach_dtx_threshold = 200;
pucch0_dtx_threshold = 150;
ofdm_offset_divisor = 8; #set this to UINT_MAX for offset 0
}
);
RUs = (
{
local_rf = "yes"
nb_tx = 1
nb_rx = 1
att_tx = 0
att_rx = 0;
bands = [78];
max_pdschReferenceSignalPower = -27;
max_rxgain = 114;
eNB_instances = [0];
#beamforming 1x4 matrix:
bf_weights = [0x00007fff, 0x0000, 0x0000, 0x0000];
clock_src = "internal";
}
);
THREAD_STRUCT = (
{
#three config for level of parallelism "PARALLEL_SINGLE_THREAD", "PARALLEL_RU_L1_SPLIT", or "PARALLEL_RU_L1_TRX_SPLIT"
parallel_config = "PARALLEL_SINGLE_THREAD";
#two option for worker "WORKER_DISABLE" or "WORKER_ENABLE"
worker_config = "WORKER_ENABLE";
}
);
rfsimulator: {
serveraddr = "server";
serverport = 4043;
options = (); #("saviq"); or/and "chanmod"
modelname = "AWGN";
IQfile = "/tmp/rfsimulator.iqs"
}
log_config: {
...
...
...
};
Helm Step Installation
Helm Configuration of OAI-CU and OAI-DU
Very Important : To access internet in NR-UE the N6/SGI interface of UPF should be able access the internet.
OAI-CU requires the ip-address or service name of AMF. In case in AMF multus is used and N1/N2 interface is bind to multus interface, then please provide AMF ip-address.
Deploy OAI-CU
cd user_n/oai-e2e/
helm install oai-cu-user-n oai-cu/ --namespace user-n
Check if the oai-cu
is started
kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=oai-cu-user-n --namespace user-n --timeout=3m
Deploy OAI-DU (RFSimulator)
cd user_n/oai-e2e/
helm install oai-du-user-n oai-du/ --namespace user-n
Check if the oai-du
is started
kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=oai-du-user-n --namespace user-n --timeout=3m
Validation CU –> DU Connection
To check if oai-du is connected to oai-cu, read the logs of cu and check F1 setup procedure was correct or not
kubectl logs --namespace user-n $(kubectl get pods --namespace user-n | grep oai-cu | awk '{print $1}') | grep 'Received F1 Setup Request'
838801.974035 [NR_RRC] I Received F1 Setup Request from gNB_DU 3584 (oai-du-rfsim) on assoc_id 189
Validation CU –> AMF Connection
To check if oai-cu is connected to amf, read the logs of amf and check N2 setup procedure was correct or not,
kubectl logs --namespace core-network $(kubectl get pods --namespace core-network | grep oai-amf| awk '{print $1}') | grep 'Connected'
Defaulted container "amf" out of: amf, init (init)
[2023-12-12 10:49:54.879] [amf_app] [info] | 1 | Connected | 0xe000 | oai-cu | 001, 01 |
[2023-12-12 10:50:14.880] [amf_app] [info] | 1 | Connected | 0xe000 | oai-cu | 001, 01 |
[2023-12-12 10:50:34.880] [amf_app] [info] | 1 | Connected | 0xe000 | oai-cu | 001, 01 |
[2023-12-12 10:50:54.880] [amf_app] [info] | 1 | Connected | 0xe000 | oai-cu | 001, 01 |
[2023-12-12 10:51:14.880] [amf_app] [info] | 1 | Connected | 0xe000 | oai-cu | 001, 01 |
Deploy OAI-NR-UE RFSimulator
Configure the UE Chart
Do helm install to start the deployment
cd user_n/oai-e2e/
helm install nr-ue oai-nr-ue-user-n/ --namespace oai-gnb-ue
Check if the oai-nr-ue
is started
kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=oai-nr-ue-user-n --timeout=3m --namespace user-n
Validation UE got IP Address
Now you are start reading the logs of amf, smf and other network function to understand the message flow. Once the pdu session establishment procedure is finished you will receive an ip-address. You can start performing some testing.
check if the UE received an ip-address
kubectl exec -it -n user-n -c nr-ue $(kubectl get pods -n user-n | grep oai-nr-ue-user-n | awk '{print $1}') -- ifconfig oaitun_ue1 | grep -E '(^|\s)inet($|\s)' | awk '{print $2}'
12.1.1.100
Performing E2E traffic testing
Inside the nr-ue pod there is an extra tcdump container which can be use to perform traffic testing via iperf3 or
kubectl exec -it -n user-n -c nr-ue $(kubectl get pods -n user-n | grep oai-nr-ue-user-n | awk '{print $1}') -- ping -I oaitun_ue1 -c4 google.fr
PING google.fr (216.58.213.67) from 12.1.1.100 oaitun_ue1: 56(84) bytes of data.
64 bytes from par21s18-in-f3.1e100.net (216.58.213.67): icmp_seq=1 ttl=117 time=27.0 ms
64 bytes from par21s18-in-f3.1e100.net (216.58.213.67): icmp_seq=2 ttl=117 time=22.3 ms
64 bytes from par21s18-in-f3.1e100.net (216.58.213.67): icmp_seq=3 ttl=117 time=24.1 ms
64 bytes from par21s18-in-f3.1e100.net (216.58.213.67): icmp_seq=4 ttl=117 time=22.7 ms
--- google.fr ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3003ms
rtt min/avg/max/mdev = 22.375/24.072/27.031/1.833 ms
## incase above doesn't work try with 8.8.8.8 instead of dns. If that works then probably you have't configure dns properly in SMF.
This part is where I validate the end-to-end control plane connection between the OAI RAN components.
Testing Backhaul (NGAP)
SCTP Socket
Validate the port on CU pod, using netstat -Spn
Wireshark
Testing Midhaul (F1AP)
SCTP Socket
Wireshark
Testing Fronthaul
This build use simulator mode of the oai/openairinterface5g
so one way to make sure that the fronthaul setup is correct is to check the port 4043 is on the LISTEN
state and will be used by UE to connect into the gNB.
Parameter configured in gNB.du.conf
#...
rfsimulator: {
serveraddr = "server";
serverport = "4043";
options = (); #("saviq"); or/and "chanmod"
modelname = "AWGN";
IQfile = "/tmp/rfsimulator.iqs"
}
#...
Testing Air Interface (Simulator)
nr-softmodem
TCP Socket
The air interface connection between UE and RFsim is emulated through tcp socket, that's why you will have an established tcp connection between the nr-softmodem
and nr-uesoftmodem
if your RFsim is connected properly with the UEsim.
UE Attach Request & Response
An end-to-end connection of 5G setup should shows a successfull PDU and GTP tunnel traffic.
RFSimulator
PDU Session
A success attach procedure should shows that the core network successfully finished the UEContextSetupRequest
from UE and give UE dataplane IP.
GTP Tunnel
Do routing in UE and UPF in the first place
UPF Routing Configuration
UE Routing Configuration
Ping test from UE simulator to UPF node through GTP Tunnel.
Captured gtp traffic In Wireshark.
nrRRC_stats.log
| Unattached UEnrRRC_stats.log
| Attached UEnrL1_stats.log
nrMAC_stats.log
nrL1_UE_stats-0.log
nrL1_UE_stats-0.log
Effective Resource Planning Usage for Users (NS Limit) w/ @arigints
Core Network
ari@node3:~$ k top pods -n core-network
NAME CPU(cores) MEMORY(bytes)
basic-mysql-569dbf6959-jcbtx 5m 193Mi
oai-amf-d8ff8dd4c-fb5qh 19m 26Mi
oai-ausf-8568fbdbc4-44pmx 30m 19Mi
oai-nrf-5754745f69-ddbdw 36m 18Mi
oai-smf-7897876bbc-27rnl 11m 55Mi
oai-udm-fd4cfc587-gwwbp 33m 20Mi
oai-udr-545ff7fb6d-jpl4w 32m 24Mi
oai-upf-d66f7b659-5vsmd 10m 196Mi
phpmyadmin-deployment-658867877d-5mh8x 1m 68Mi
Single
ari@node3:~$ k top pods -n user-n
NAME CPU(cores) MEMORY(bytes)
oai-cu-6b96c97dd7-987st 1m 116Mi
oai-du-795877db8b-kpggd 1577m 1303Mi
oai-nr-ue-76c9866b57-xf668 1215m 533Mi
Multi DU
ari@node3:~$ k top pods -n user-n
NAME CPU(cores) MEMORY(bytes)
oai-cu-mdu-cf7c86df-q7q7m 3m 116Mi
oai-du-mdu-1-5f4b59779f-vvzrg 1597m 1303Mi
oai-du-mdu-2-979d59d77-92zl5 1591m 1303Mi
oai-nr-ue-mdu-1-679cd477b6-7xwtb 966m 530Mi
oai-nr-ue-mdu-2-b77bf7d9b-nwkwd 952m 530Mi
Multi UE
ari@node3:~$ k top pods -n user-n
NAME CPU(cores) MEMORY(bytes)
oai-cu-mue-7756669d95-dlql7 14m 116Mi
oai-du-mue-976b4cf78-bp8g5 1365m 1309Mi
oai-nr-ue-mue-1-6779c8cb54-vpc5t 646m 530Mi
oai-nr-ue-mue-2-69878cc8b-fzmt8 672m 529Mi
Raw Calculation
Core Network CPU and Memory Utilization
First, we'll sum up the CPU and memory usage for all the components in the Core Network:
Total CPU (m): 5+19+30+36+11+33+32+10+1=177m
Total Memory (Mi): 193+26+19+18+55+20+24+196+68=619Mi
Next, we'll calculate the percentage of CPU and memory each component uses relative to these totals.
Single CPU and Memory Utilization
For the Single category:
Total CPU (m): 1+1577+1215=2793m
Total Memory (Mi): 116+1303+533=1952Mi
Multi DU CPU and Memory Utilization
For the Multi DU category:
Total CPU (m): 3+1597+1591+966+952=5109m
Total Memory (Mi): 116+1303+1303+530+530=3782Mi
Multi UE CPU and Memory Utilization
For the Multi UE category:
Total CPU (m): 14+1365+646+672=2697m
Total Memory (Mi): 116+1309+530+529=2484Mi
Planning for Reducing Resource
CU Configuration
config:
mountConfig: true #If config file is mounted then please edit mount.conf in configmap.yaml properly
timeZone: "Asia/Jakarta"
useAdditionalOptions: "--sa --log_config.global_log_options level,nocolor,time"
# If mounting the configuration file then below parameters are not used
cuName: "oai-cu"
cuId: "0xe04"
cellId: "12345678L"
mcc: "208" # check the information with AMF, SMF, UPF
mnc: "99" # check the information with AMF, SMF, UPF
tac: "0xa001" # check the information with AMF
sst: "1" #currently only 4 standard values are allowed 1,2,3,4
amfhost: "172.21.6.94" # amf ip-address or service-name oai-amf-svc or 172.21.6.94
n2IfName: "n2" # if multus.n2Interface.create is true then use n2
n3IfName: "n3" #if multus.n3Interface.create is true then use n3 or you can only use 1 interface n2 or eth0
f1IfName: "f1" #if multus.f1Interface.create is true then use multus.f1Interface.Ipadd
f1cuPort: "2152" #2153 if using same interface for f1 and n3 else standard port 2152 should be use if f1 and n3 interface are different
f1duPort: "2152" #2153 if using same interface for f1 and n3 else standard port 2152 should be use if f1 and n3 interface are different
DU Configuration
config:
mountConfig: true #If config file is mounted then please edit mount.conf in templates/configmap.yaml properly
timeZone: "Asia/Jakarta"
useAdditionalOptions: "--sa --rfsim -r 106 --numerology 1 -C 3619200000 --nokrnmod --log_config.global_log_options level,nocolor,time"
gnbId: "0xe05"
duName: "oai-du-rfsim"
duId: "0xe06"
cellId: "12345678L"
mcc: "208" # check the information with AMF, SMF, UPF
mnc: "99" # check the information with AMF, SMF, UPF
tac: "0xa001" # check the information with AMF
sst: "1" #currently only 4 standard values are allowed 1,2,3,4
usrp: rfsim #allowed values rfsim, b2xx, n3xx or x3xx
f1IfName: "f1" #if multus.f1Interface.create is true then use f1
cuHost: "192.168.1.1" ## Ip-address or hostname
f1cuPort: "2152" #2153 if using same interface for f1 and n3 else standard port 2152 should be use if f1 and n3 interface are different
f1duPort: "2152" #2153 if using same interface for f1 and n3 else standard port 2152 should be use if f1 and n3 interface are different
RU Configuration
////////// Physical parameters:
min_rxtxtime = 6;
force_256qam_off = 1;
servingCellConfigCommon = (
{
#spCellConfigCommon
physCellId = 0;
# downlinkConfigCommon
#frequencyInfoDL
# this is 3600 MHz + 43 PRBs@30kHz SCS (same as initial BWP)
absoluteFrequencySSB = 641280;
dl_frequencyBand = 78;
# this is 3600 MHz
dl_absoluteFrequencyPointA = 640008;
#scs-SpecificCarrierList
dl_offstToCarrier = 0;
# subcarrierSpacing
# 0=kHz15, 1=kHz30, 2=kHz60, 3=kHz120
dl_subcarrierSpacing = 1;
dl_carrierBandwidth = 106;
#initialDownlinkBWP
#genericParameters
# this is RBstart=27,L=48 (275*(L-1))+RBstart
initialDLBWPlocationAndBandwidth = 28875; # 6366 12925 12956 28875 12952
# subcarrierSpacing
# 0=kHz15, 1=kHz30, 2=kHz60, 3=kHz120
initialDLBWPsubcarrierSpacing = 1;
#pdcch-ConfigCommon
initialDLBWPcontrolResourceSetZero = 12;
initialDLBWPsearchSpaceZero = 0;
#uplinkConfigCommon
#frequencyInfoUL
ul_frequencyBand = 78;
#scs-SpecificCarrierList
ul_offstToCarrier = 0;
# subcarrierSpacing
# 0=kHz15, 1=kHz30, 2=kHz60, 3=kHz120
ul_subcarrierSpacing = 1;
ul_carrierBandwidth = 106;
pMax = 20;
#initialUplinkBWP
#genericParameters
initialULBWPlocationAndBandwidth = 28875;
# subcarrierSpacing
# 0=kHz15, 1=kHz30, 2=kHz60, 3=kHz120
initialULBWPsubcarrierSpacing = 1;
#rach-ConfigCommon
#rach-ConfigGeneric
prach_ConfigurationIndex = 98;
#prach_msg1_FDM
#0 = one, 1=two, 2=four, 3=eight
prach_msg1_FDM = 0;
prach_msg1_FrequencyStart = 0;
zeroCorrelationZoneConfig = 13;
preambleReceivedTargetPower = -96;
#preamblTransMax (0...10) = (3,4,5,6,7,8,10,20,50,100,200)
preambleTransMax = 6;
#powerRampingStep
# 0=dB0,1=dB2,2=dB4,3=dB6
powerRampingStep = 1;
#ra_ReponseWindow
#1,2,4,8,10,20,40,80
ra_ResponseWindow = 4;
#ssb_perRACH_OccasionAndCB_PreamblesPerSSB_PR
#1=oneeighth,2=onefourth,3=half,4=one,5=two,6=four,7=eight,8=sixteen
ssb_perRACH_OccasionAndCB_PreamblesPerSSB_PR = 4;
#oneHalf (0..15) 4,8,12,16,...60,64
ssb_perRACH_OccasionAndCB_PreamblesPerSSB = 14;
#ra_ContentionResolutionTimer
#(0..7) 8,16,24,32,40,48,56,64
ra_ContentionResolutionTimer = 7;
rsrp_ThresholdSSB = 19;
#prach-RootSequenceIndex_PR
#1 = 839, 2 = 139
prach_RootSequenceIndex_PR = 2;
prach_RootSequenceIndex = 1;
# SCS for msg1, can only be 15 for 30 kHz < 6 GHz, takes precendence over the one derived from prach-ConfigIndex
#
msg1_SubcarrierSpacing = 1,
# restrictedSetConfig
# 0=unrestricted, 1=restricted type A, 2=restricted type B
restrictedSetConfig = 0,
msg3_DeltaPreamble = 1;
p0_NominalWithGrant =-90;
# pucch-ConfigCommon setup :
# pucchGroupHopping
# 0 = neither, 1= group hopping, 2=sequence hopping
pucchGroupHopping = 0;
hoppingId = 40;
p0_nominal = -90;
# ssb_PositionsInBurs_BitmapPR
# 1=short, 2=medium, 3=long
ssb_PositionsInBurst_PR = 2;
ssb_PositionsInBurst_Bitmap = 1;
# ssb_periodicityServingCell
# 0 = ms5, 1=ms10, 2=ms20, 3=ms40, 4=ms80, 5=ms160, 6=spare2, 7=spare1
ssb_periodicityServingCell = 2;
# dmrs_TypeA_position
# 0 = pos2, 1 = pos3
dmrs_TypeA_Position = 0;
# subcarrierSpacing
# 0=kHz15, 1=kHz30, 2=kHz60, 3=kHz120
subcarrierSpacing = 1;
#tdd-UL-DL-ConfigurationCommon
# subcarrierSpacing
# 0=kHz15, 1=kHz30, 2=kHz60, 3=kHz120
referenceSubcarrierSpacing = 1;
# pattern1
# dl_UL_TransmissionPeriodicity
# 0=ms0p5, 1=ms0p625, 2=ms1, 3=ms1p25, 4=ms2, 5=ms2p5, 6=ms5, 7=ms10
dl_UL_TransmissionPeriodicity = 6;
nrofDownlinkSlots = 7;
nrofDownlinkSymbols = 6;
nrofUplinkSlots = 2;
nrofUplinkSymbols = 4;
ssPBCH_BlockPower = -25;
}
);
UE Configuration
config:
timeZone: "Asia/Jakarta"
rfSimServer: "192.168.1.2" # ip-address of rfsim or service name oai-gnb or oai-du
fullImsi: "208990100001101" # make sure all the below entries are present in the subscriber database
fullKey: "a92d71ed92f43ebae1720f34fd2e6966"
opc: "C42449363BBAD02B66D16BC975D77CC1"
dnn: "oai"
sst: "1" # configure according to gnb and amf, smf and upf
sd: "FFFFF4"
usrp: "rfsim" # allowed rfsim, b2xx, n3xx, x3xx
useAdditionalOptions: "--sa --rfsim -r 106 --numerology 1 -C 3619200000 --nokrnmod --log_config.global_log_options level,nocolor,time"
System Topologi
Each user has 3 stages. First they have to configure the Single DU & UE, while the other stages are down (Replicaset equal to 0). Then after all the parameters on stages 1 complete, users can go to the next stages to complete the training.
Subnet IP for User N
Multi DU
Purpose | Interface | IP Address | Port |
---|---|---|---|
DU 1 | F1 | 172.21.6.90/22 | 38472 |
DU 2 | F1 | 172.21.8.91/22 | 2152 |
UE 1 | Net1 | 192.168.1.5/27 | –- |
UE 2 | Net1 | 192.168.1.5/27 | –- |
Multi UE
Purpose | Interface | IP Address | Port |
---|---|---|---|
DU 1 | F1 | 172.21.6.90/22 | 38472 |
UE 1 | Net1 | 192.168.1.5/27 | –- |
UE 2 | Net1 | 192.168.1.5/27 | –- |
Restriction IP to Access
And using /28
block IP per users. Or using VLAN but still TBD.
Helm Step Installation
Helm Configuration of OAI-CU and OAI-DU
Very Important : To access internet in NR-UE the N6/SGI interface of UPF should be able access the internet.
OAI-CU requires the ip-address or service name of AMF. In case in AMF multus is used and N1/N2 interface is bind to multus interface, then please provide AMF ip-address.
Deploy OAI-CU
cd user_n/oai-e2e/
helm install oai-cu-user-n oai-cu/ --namespace user-n
Check if the oai-cu
is started
kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=oai-cu-user-n --namespace user-n --timeout=3m
Deploy OAI-DU (RFSimulator)
cd user_n/oai-e2e/
helm install oai-du-user-n oai-du/ --namespace user-n
Check if the oai-du
is started
kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=oai-du-user-n --namespace user-n --timeout=3m
Validation CU –> DU Connection
To check if oai-du is connected to oai-cu, read the logs of cu and check F1 setup procedure was correct or not
kubectl logs --namespace user-n $(kubectl get pods --namespace user-n | grep oai-cu | awk '{print $1}') | grep 'Received F1 Setup Request'
838801.974035 [NR_RRC] I Received F1 Setup Request from gNB_DU 3584 (oai-du-rfsim) on assoc_id 189
Validation CU –> AMF Connection
To check if oai-cu is connected to amf, read the logs of amf and check N2 setup procedure was correct or not,
kubectl logs --namespace core-network $(kubectl get pods --namespace core-network | grep oai-amf| awk '{print $1}') | grep 'Connected'
Defaulted container "amf" out of: amf, init (init)
[2023-12-12 10:49:54.879] [amf_app] [info] | 1 | Connected | 0xe000 | oai-cu | 001, 01 |
[2023-12-12 10:50:14.880] [amf_app] [info] | 1 | Connected | 0xe000 | oai-cu | 001, 01 |
[2023-12-12 10:50:34.880] [amf_app] [info] | 1 | Connected | 0xe000 | oai-cu | 001, 01 |
[2023-12-12 10:50:54.880] [amf_app] [info] | 1 | Connected | 0xe000 | oai-cu | 001, 01 |
[2023-12-12 10:51:14.880] [amf_app] [info] | 1 | Connected | 0xe000 | oai-cu | 001, 01 |
Deploy OAI-NR-UE RFSimulator
Configure the UE Chart
Do helm install to start the deployment
cd user_n/oai-e2e/
helm install nr-ue oai-nr-ue-user-n/ --namespace oai-gnb-ue
Check if the oai-nr-ue
is started
kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=oai-nr-ue-user-n --timeout=3m --namespace user-n
Validation UE got IP Address
Now you are start reading the logs of amf, smf and other network function to understand the message flow. Once the pdu session establishment procedure is finished you will receive an ip-address. You can start performing some testing.
check if the UE received an ip-address
kubectl exec -it -n user-n -c nr-ue $(kubectl get pods -n user-n | grep oai-nr-ue-user-n | awk '{print $1}') -- ifconfig oaitun_ue1 | grep -E '(^|\s)inet($|\s)' | awk '{print $2}'
12.1.1.100
Performing E2E traffic testing
Inside the nr-ue pod there is an extra tcdump container which can be use to perform traffic testing via iperf3 or
kubectl exec -it -n user-n -c nr-ue $(kubectl get pods -n user-n | grep oai-nr-ue-user-n | awk '{print $1}') -- ping -I oaitun_ue1 -c4 google.fr
PING google.fr (216.58.213.67) from 12.1.1.100 oaitun_ue1: 56(84) bytes of data.
64 bytes from par21s18-in-f3.1e100.net (216.58.213.67): icmp_seq=1 ttl=117 time=27.0 ms
64 bytes from par21s18-in-f3.1e100.net (216.58.213.67): icmp_seq=2 ttl=117 time=22.3 ms
64 bytes from par21s18-in-f3.1e100.net (216.58.213.67): icmp_seq=3 ttl=117 time=24.1 ms
64 bytes from par21s18-in-f3.1e100.net (216.58.213.67): icmp_seq=4 ttl=117 time=22.7 ms
--- google.fr ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3003ms
rtt min/avg/max/mdev = 22.375/24.072/27.031/1.833 ms
## incase above doesn't work try with 8.8.8.8 instead of dns. If that works then probably you have't configure dns properly in SMF.
Helm Step Installation
Helm Configuration of OAI-CU and OAI-DU
Very Important : To access internet in NR-UE the N6/SGI interface of UPF should be able access the internet.
OAI-CU requires the ip-address or service name of AMF. In case in AMF multus is used and N1/N2 interface is bind to multus interface, then please provide AMF ip-address.
Deploy OAI-CU
cd user_n/oai-e2e/
helm install oai-cu-user-n oai-cu/ --namespace user-n
Check if the oai-cu
is started
kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=oai-cu-user-n --namespace user-n --timeout=3m
Deploy OAI-DU (RFSimulator)
cd user_n/oai-e2e/
helm install oai-du-user-n oai-du/ --namespace user-n
Check if the oai-du
is started
kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=oai-du-user-n --namespace user-n --timeout=3m
Validation CU –> DU Connection
To check if oai-du is connected to oai-cu, read the logs of cu and check F1 setup procedure was correct or not
kubectl logs --namespace user-n $(kubectl get pods --namespace user-n | grep oai-cu | awk '{print $1}') | grep 'Received F1 Setup Request'
838801.974035 [NR_RRC] I Received F1 Setup Request from gNB_DU 3584 (oai-du-rfsim) on assoc_id 189
Validation CU –> AMF Connection
To check if oai-cu is connected to amf, read the logs of amf and check N2 setup procedure was correct or not,
kubectl logs --namespace core-network $(kubectl get pods --namespace core-network | grep oai-amf| awk '{print $1}') | grep 'Connected'
Defaulted container "amf" out of: amf, init (init)
[2023-12-12 10:49:54.879] [amf_app] [info] | 1 | Connected | 0xe000 | oai-cu | 001, 01 |
[2023-12-12 10:50:14.880] [amf_app] [info] | 1 | Connected | 0xe000 | oai-cu | 001, 01 |
[2023-12-12 10:50:34.880] [amf_app] [info] | 1 | Connected | 0xe000 | oai-cu | 001, 01 |
[2023-12-12 10:50:54.880] [amf_app] [info] | 1 | Connected | 0xe000 | oai-cu | 001, 01 |
[2023-12-12 10:51:14.880] [amf_app] [info] | 1 | Connected | 0xe000 | oai-cu | 001, 01 |
Deploy OAI-NR-UE RFSimulator
Configure the UE Chart
Do helm install to start the deployment
cd user_n/oai-e2e/
helm install nr-ue oai-nr-ue-user-n/ --namespace oai-gnb-ue
Check if the oai-nr-ue
is started
kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=oai-nr-ue-user-n --timeout=3m --namespace user-n
Validation UE got IP Address
Now you are start reading the logs of amf, smf and other network function to understand the message flow. Once the pdu session establishment procedure is finished you will receive an ip-address. You can start performing some testing.
check if the UE received an ip-address
kubectl exec -it -n user-n -c nr-ue $(kubectl get pods -n user-n | grep oai-nr-ue-user-n | awk '{print $1}') -- ifconfig oaitun_ue1 | grep -E '(^|\s)inet($|\s)' | awk '{print $2}'
12.1.1.100
Performing E2E traffic testing
Inside the nr-ue pod there is an extra tcdump container which can be use to perform traffic testing via iperf3 or
kubectl exec -it -n user-n -c nr-ue $(kubectl get pods -n user-n | grep oai-nr-ue-user-n | awk '{print $1}') -- ping -I oaitun_ue1 -c4 google.fr
PING google.fr (216.58.213.67) from 12.1.1.100 oaitun_ue1: 56(84) bytes of data.
64 bytes from par21s18-in-f3.1e100.net (216.58.213.67): icmp_seq=1 ttl=117 time=27.0 ms
64 bytes from par21s18-in-f3.1e100.net (216.58.213.67): icmp_seq=2 ttl=117 time=22.3 ms
64 bytes from par21s18-in-f3.1e100.net (216.58.213.67): icmp_seq=3 ttl=117 time=24.1 ms
64 bytes from par21s18-in-f3.1e100.net (216.58.213.67): icmp_seq=4 ttl=117 time=22.7 ms
--- google.fr ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3003ms
rtt min/avg/max/mdev = 22.375/24.072/27.031/1.833 ms
## incase above doesn't work try with 8.8.8.8 instead of dns. If that works then probably you have't configure dns properly in SMF.