This guideline explains how to configure and change the setting of UPF to work either formally with Open5gs or individually with T-Rex.
# UPF settings:
The settings of UPF is designed to work with 3 interfaces:
* N4: belong to control plane which communicates with SMF function in 5G core network. The N4 interface employs the PFCP protocol to establish PFCP sessions which defines how packets are identified (Packet Detection Rule), forwarded (Forwarding Action Rules), processed (Buffering Action Rules), marked (QoS Enforcement Rules) and reported (Usage Reporting Rules).
* N3: belong to user plane, connect the gNodeB to the UPF. The interface conveys user data to the UPF for processing
- Uplink: N3 receives packet which was encapsulated by gNodeB using GTP-U protocol before decapsulated by UPF and forwarded to N6 interface.
- Downlink: N3 receives packet encapsulated by UPF using GTP-U protocol after received from datanetwork (DN) through N6 interface and then transmit to gNodeB.
* N6: belong to user plane, which has the duty to send/receive packet UPF to/from DN.
## Initialization:
The N3 and N6 interface is intended to use with DPDK interface to fully utilize the fast-path functionality. For the Adlink-MECS, binding the SFP port 2 & 3 with DPDK driver by:

sudo dpdk-devbind.py -b igb_uio b8:00.2
sudo dpdk-devbind.py -b igb_uio b8:00.3

The N4 interface is used for management purpose and it is binded with the physical "enp3s0" interface. A virtual bridge "virbr0" is created by the operating system and attached to "enp3s0" so we can use it to bridge "enp3s0" with UPF:

sudo ifconfig enp3s0 up
sudo ip link add name n4_br type veth peer name n4
sudo brctl addif virbr0 n4_br
sudo ip addr add 192.168.74.112/24 dev n4_br
sudo ip link set dev n4_br up
sudo ip link set dev n4 up

### N4 IP address settings:
The N4 inside UPF associated with the "n4" created in the initialization step above is created with name "host-n4".

create host-interface name n4
set interface mtu 1500 host-n4
set interface ip table host-n4 0
set interface ip address host-n4 192.168.1.112/24
set interface state host-n4 up

### N3 IP settings:
The N3 interface inside UPF is associated directly with the 10G SFP physical interface (PCI address b8:00.3), which is controlled by DPDK driver now, so we can directly assign its IP settings:

set interface ip table TenGigabitEthernetb8/0/3 2
set interface mtu 9000 TenGigabitEthernetb8/0/3
set interface ip address TenGigabitEthernetb8/0/3 192.168.73.118/24
set interface state TenGigabitEthernetb8/0/3 up

### N6 IP settings:
Similarly to N3, the IP settings for N6 interface inside UPF associated with the SFP interface (PCI address b8:00.2) is shown below:

set interface ip table TenGigabitEthernetb8/0/2 1
set interface mtu 9000 TenGigabitEthernetb8/0/2
set interface ip address TenGigabitEthernetb8/0/2 192.168.76.118/24
set interface state TenGigabitEthernetb8/0/2 up

# Set-up with Open5gs

digraph G {
  newrank=true;
  rankdir="LR"
  subgraph cluster_0 {
      label="UERANSIM";      
      xlabel="192.168.1.242";
      UE [label = "UE";shape = rect;];
      gNB [label = "gNB";shape = rect;];
  } 
  subgraph cluster_1 {
     label="Open5gs";
     xlabel="a"
     AMF [ label = "AMF"; shape = rect;];
     SMF [label = "SMF";shape = rect;];
  }
  subgraph cluster_2 {
    N4 [label = "N4";shape = rect; ];
    N3 [label = "N3";shape = rect; ];   
    N6 [label = "N6";shape = rect;];
    label="UPF"; color = red; 
    edge [style=invis]
  }
  DN [label = "DN";shape =rect;]
  UE -> gNB [dir="both";color= blue; style=dashed];
  AMF -> gNB [label = "NGAP/NAS";dir="both";];
  SMF -> N4 [label = "PFCP";dir="both";];
  gNB -> N3 [label = "GTP-U"; dir="both";color = magenta];
  N6 -> DN [label = "IP"; dir="both";color = magenta];
  
  gNB -> N3 -> N6 -> DN [style = invis]
  #N3 -> N6 [ style = dashed; dir="both"; color = green]
  {rank = same; gNB; UE}
  #{rank = same; AMF; gNB }
  #{rank = same; SMF; N4}
  
}

## Open5gs settings
The 5G Core Open5gs is configured to use our UPF instead of the default UPF, hence it is necessary to make some change in the Open5gs settings. The Open5gs is installed at a virtual machine @192.168.1.160.
### AMF

amf:
    sbi:
      - addr: 127.0.0.5
        port: 7777
    ngap:
      - addr: 192.168.1.160
    metrics:
      - addr: 127.0.0.5
        port: 9090
    guami:
      - plmn_id:
          mcc: 208
          mnc: 95
        amf_id:
          region: 2
          set: 1
    tai:
      - plmn_id:
          mcc: 208
          mnc: 95
        tac: 1
    plmn_support:
      - plmn_id:
          mcc: 208
          mnc: 95
        s_nssai:
          - sst: 1
    security:
        integrity_order : [ NIA2, NIA1, NIA0 ]
        ciphering_order : [ NEA0, NEA1, NEA2 ]
    network_name:
        full: Open5GS
    amf_name: open5gs-amf0

### SMF

smf:
    sbi:
      - addr: 127.0.0.4
        port: 7777
    pfcp:
      - addr: 192.168.1.160
      - addr: ::1
    gtpc:
      - addr: 127.0.0.4
      - addr: ::1
    gtpu:
      - addr: 192.168.1.160 
      - addr: ::1
    metrics:
      - addr: 127.0.0.4
        port: 9090
    subnet:
      - addr: 10.45.0.1/16
      - addr: 2001:db8:cafe::1/48
    dns:
      - 8.8.8.8
      - 8.8.4.4
      - 2001:4860:4860::8888
      - 2001:4860:4860::8844
    mtu: 1400
    ctf:
      enabled: auto
    freeDiameter: /home/luser/open5gs/install/etc/freeDiameter/smf.conf

upf:
    pfcp:
      - addr: 192.168.1.112 

## UPF routing table
In order to receive and forward packet from/to Open5gs (N4), gNodeB (N3) and DN(N6), we need to configure the routing table used by the UPF. It is done as below:

ip route add 0.0.0.0/0 table 0 via 192.168.1.160 host-n4
ip route add 0.0.0.0/0 table 2 via 192.168.73.117 TenGigabitEthernetb8/0/3
ip route add 0.0.0.0/0 table 1 via 192.168.76.138 TenGigabitEthernetb8/0/2

## DN routing table
The DN here is binded to a bare-metal PC which can ssh@192.168.1.17. One of its SFP interface ("enp1s0f1") is connected with the UPF's N6 interface. To forward data packet from user plane to DN successfully, use the following commands:

sudo ip addr add 192.168.76.138/24 dev enp1s0f1
sudo iptables -t nat -A POSTROUTING -o enp1s0f1 -j MASQUERADE
sudo ip route add 10.45.0.0/24 via 192.168.76.118 dev enp1s0f1

## UERANSIM settings
We use UERANSIM to simulate gNodeB and UE. This package is installed at the server Dell R740@192.168.1.242. We configure two physical interfaces "eno1" and "ens4f0" as N2 (between gNodeB and AMF), N3 (between gNodeB and UPF) respectively. The "eno1" interface is already configured with IP 192.168.1.242 by default. Issue the following command to configure the IP address for the physical interface "ens4f0" as N3:

`sudo ip addr add 192.168.73.117/24 dev ens4f0`

### gNodeB
To run gNodeB, run the command

cd UERANSIM
./build/nr-gnb -c config/open5gs-gnb-vpp-upf-x86.yaml

### UE
To control a UE to attach to gNodeB and have registration to 5G core, run the command

cd UERANSIM
sudo ./build/nr-ue -c config/open5gs-ue-gv.yaml

# Set-up with TRex

digraph G1 {
  newrank=true;
  rankdir = "LR";
       
  PFCP [ label = "PFCP-Sim"; shape = rect;];
  subgraph cluster_5 {
      label="TRex";
      P0 [label = "Port0";shape = rect;];
      P1 [label = "Port1";shape = rect;];
  }
   
  subgraph cluster_6 {
      N3_T [label = "N3";shape = rect;];
      N4_T [label = "N4";shape = rect;];
      N6_T [label = "N6";shape = rect;];
      label="UPF"; color = red; 
      edge [style=invis]
  }
 
  PFCP -> N4_T [label = "PFCP";dir="both";];
  P1 -> N6_T [label = "IP";dir="both";color = magenta]
  P0 -> N3_T [label = "GTP-U"; dir="both";color = magenta]
  N3_T -> N6_T [style =invis] #;dir="both";color = green]
  
  
  {rank = same; N3_T; P0; }
  {rank = same; N6_T; P1;}
  {rank = same; N3_T; N4_T;}
  {rank = same; N4_T;  }
}

TRex is a powerful tool to test the upper bound performance of our UPF without setting up the entire system i.e. with gNodeB/UE and core network. However, it is still needed to establish PFCP sessions to configure UPF to process incoming packets from/to N3 and N6. We use
"pfcp-kitchen-sink" tool as a PFCP simulator.

## Initialization
We install TRex into the same bare-metal PC used for DN. We need to prepare 2 physical interfaces named Port0 and Port1. These ports will connect to the N3/N6 interface by cables in the UPF respectively. TRex will automatically manage them with DPDK driver once the TRex server runs. However, before running the TRex server, we use the following commands to configure the 2 SFP interfaces at PCI address 0 and 1 and initialize them with IP addressed later used by TRex.

cd /opt/trex/v3.03
sudo ./dpdk_nic_bind.py -b i40e 01:00.0
sudo ./dpdk_nic_bind.py -b i40e 01:00.1
sudo ip addr add 192.168.73.17/24 dev enp1s0f0
sudo ip addr add 192.168.76.137/24 dev enp1s0f1

## PFCP-Sim
We installed "pfcp-kitchen-sink" at the bare-metal PC at 192.168.1.17. Remember that we have configured the N4 interface of the UPF with IP address 192.168.1.112. Therefore, the pfcp simulator will need to associate with this UPF's N4 to establish PFCP sessions. Run the following command:

cd /home/luser/pfcp-kitchen-sink
./pfcpclient -r 192.168.1.112:8805 -s examples/sessions-vppupf-AdlinkUPF.yaml

## TRex
After the PFCP sessions are successfully established, run the TRex server by:
`/opt/trex/v3.03$ sudo ./t-rex-64 -i -c 1`
Open another terminal and run the TRex console where we can run different tests:
`/opt/trex/v3.03$ sudo ./trex-console`

Once the Trex console is connected successfully with the TRex server, we can test the UPF performace under two scenarios, (i) uplink: packets are sent from UE to DN and (ii) downlink: packet are sent from DN to UE.

* Uplink

Since the pfcp-sim can only establish the PFCP session with the UPF through N4 interface, we need to modify the TRex's code to generate packet in the same way as generated by the gNodeB.
For illustration purpose, we take the ICMP packet as an example of how it is modified to be exactly as an ICMP packet that going from gNodeB to the N3 interface of the UPF. The figure below explains the original ICMP packet.

digraph G {
  newrank=true;
  #rankdir="LR"
  EthernetHeader [label = "Ethernet Header"; shape =rect]
  IPHeader [label = "IP Header"; shape =rect]
  ICMP [label = "ICMP"; shape =rect]
  EthernetHeader->IPHeader->ICMP
  {rank = same; EthernetHeader; IPHeader; ICMP }
}

And an ICMP packet after going from gNodeB and captured at N3 interface at the UPF has the format:

digraph G {
  newrank=true;
  #rankdir="LR"
  EthernetHeader [label = "Ethernet Header"; shape =rect]
  IPHeaderOut [label = "IP Header (outer)"; shape =rect]
  UDP [label = "UDP"; shape =rect]
  GTP [label = "GTP-U Header", shape=rect]
  IPHeaderIn [label = "IP Header (inner)"; shape =rect]
  ICMP [label = "ICMP"; shape =rect]
  EthernetHeader->IPHeaderOut->UDP->GTP->IPHeaderIn->ICMP
  {rank = same; EthernetHeader; IPHeaderOut; UDP; GTP; IPHeaderIn; ICMP }
}

The following code shows how the uplink packet is modified by adding *GTP_U Header* and *GTPPDUSessionContainer* including the IP header (inner) where the IP address of the UE and Data network are kept as source and destination.

`base_pkt = Ether(dst="00:30:64:6f:c5:d3")/IP(src="192.168.73.17", dst="192.168.73.118")/UDP(dport=2152)/GTP_U_Header(teid=1234)/ GTPPDUSessionContainer(type=1, QFI=1)/IP(src="10.45.0.7",dst="192.168.76.137",version=4)/ICMP()`

To test the uplink performance, run the following command at TRex client console:

>>start -f stl/upf_bench.py -m 100% -p 0 0 -t size=1024,vm=random

* Downlink

>> start -f stl/upf_bench_dl.py -m 100% -p 1 1 -t size=1024,vm=random