Version 3：UE + gNB + multiple SMF, UPF + MEC + control plane

tags: MEC

Goal

Based on Environment setup of multiple SMF, DNN, and UPF, divide the UE and gNB into different VM and try to connect to Internet.

Result:

• UE1 (UE1 → gNB → I-UPF → PSA-UP2 → DNN:internet): failed
• UE2 (UE2 → gNB → PSA-UP1 → DNN:IMS): succeed

Topology

Troble

UE1 can' t connect Internet via tunnel

$ ping -I uesimtun0 8.8.8.8

問題排除過程

Trobleshooting

1. 首先對 PSA-UPF 手動新增靜態路由表，從 N3 interface 通往 192.168.0.0/24
   從 PSA-UPF1 去 ping UE 會發現 gNB 有收到 request，但無繼續 forward
2. 因為 gNB 已經有通往 UPF1 路由表，不必再新增

   要確認一下若此時不執行 forward command 會不會有影響

   
   ​​​​cat /proc/sys/net/ipv4/ip_forward
   ​​​​echo "1" > /proc/sys/net/ipv4/ip_forward
   

3. 接著從 gNB 的 tcpdump 會發現都是 UPF 的 arp request，且問不到 UE 的 MAC
   gNB 執行以下指令：

   
   
   
   ​​​​sudo sysctl net.ipv4.conf.all.proxy_arp=1
   ​​​​sudo sysctl net.ipv4.conf.enp0sX.proxy_arp=1 //通往 UE
   ​​​​sudo sysctl net.ipv4.conf.enp0sX.proxy_arp=1 // 通往 UPF
   ​​​​sudo sysctl -p
   

4. 嘗試從 UPF ping UE，若不行則反向測試
   此時可能就可以通了！
5. 另一條 path 照上面步驟做一次。

Configuration

UE：free5gc-ue1.yaml

• gnbSearchList： gNB Nn interface

gNB：free5gc-gnb.yaml

• linkIp： gNB Nn interface
• ngapIp： gNB N2 interface
• gtpIp： gNB N3 interface
• amfConfigs：AMF N2 interface
• sst：setting0x1 for UE2
• sd：setting 0x112233 for UE2

i-UPF：upfcfg.yaml

• pfcp：I-UPF N4 interface
• gtpu：because this UPF have N3、N9 so setting: 0.0.0.0
• dnn_list：intranet (accoarding to free5gc SMF Config / ULCL Config)
• cidr：60.60.0.0/24

PSA-UPF1：upfcfg.yaml

• pfcp：PSA-UPF1 N4
• gtpu：PSA-UPF1 N3
• dnn：IMS for UE2
• cidr：60.61.0.0/24

PSA-UPF2：upfcfg.yaml

• pcfp：PSA-UPF2 N4
• gtpu：PSA-UPF2 N9
• cidr：60.60.0.0/24

free5gc- control plane

AMF：amfcfg.yaml

• ngapIpList: AMF N2

SMF: smfcfg.yaml

• pfcp: SMF N4 for I-UPF and PSA-UPF2
• IUPF:
  • node_id: I-UPF N4
  • sNssai: for UE1
  • dnnUpfInfoList: intranet
  • interface: N3 and N9
• PSA2:
  • node_id: UPF-PSA N4
  • dnnUpfInfoList: internet
  • cidr: 60.60.0.0/24
  • interface: N9
• links: gNB1 ⇄ IUPF, IUPF ⇄ PSA2
• ulcl: true

UE routing: uerouting.yaml

• members: UE1
• topology: gNB1 ⇄ IUPF, IUPF ⇄ PSA2
• specificPath: MEC N6
• path: [IUPF,IUPF] according to Environment setup of multiple SMF, DNN, and UPF

run.sh

Steps

1. Use WebConsole to Add an UE
2. Set I-UPF networking and run：

   ​$ sudo sysctl -w net.ipv4.ip_forward=1
   ​$ sudo iptables -t nat -A POSTROUTING -o enp0s9 -j MASQUERADE
   ​$ sudo systemctl stop ufw
   ​$ sudo iptables -I FORWARD 1 -j ACCEPT
   ​$ cd ~/free5gc/NFs/upf/build/
   ​$ sudo ./bin/free5gc-upfd
   

3. Set UPF-PSA1 networking and run：

   ​$ sudo sysctl -w net.ipv4.ip_forward=1
   ​$ sudo iptables -t nat -A POSTROUTING -o enp0s3 -j MASQUERADE
   ​$ sudo systemctl stop ufw
   ​$ sudo iptables -I FORWARD 1 -j ACCEPT
   ​$ cd ~/free5gc/NFs/upf/build/
   ​$ sudo ./bin/free5gc-upfd
   

4. Set UPF-PSA2 networking and run：

   ​$ sudo sysctl -w net.ipv4.ip_forward=1
   ​$ sudo iptables -t nat -A POSTROUTING -o enp0s3 -j MASQUERADE
   ​$ sudo systemctl stop ufw
   ​$ sudo iptables -I FORWARD 1 -j ACCEPT
   ​$ cd ~/free5gc/NFs/upf/build/
   ​$ sudo ./bin/free5gc-upfd
   

5. In free5gc, run control plane

   ​$ cd free5gc
   ​$ ./run.sh
   

6. In gNB, run gNB

   ​$ cd UERANSIM
   ​$ build/nr-gnb -c config/free5gc-gnb.yaml
   

7. In UE, run UE1

   ​$ cd UERANSIM
   ​$ sudo build/nr-ue -c config/free5gc-ue1.yaml
   

8. In UE, check tunnel:uesimtun0
   

Test：UE1 ping 8.8.8.8

UE

uesimtun0

Only capture ICMP echo request (UE → 8.8.8.8) but ICMP reply (8.8.8.8 → UE)

Nn

The third packet might be ICMP request packet (UE → gNB).
I think there should appear the ICMP reply packet (gNB → UE)

gNB

N3

Nn

Second packet might be ICMP request

I-UPF

N3

N9

PSA-UPF2

NAT

N9

Test: UE1 mtr 8.8.8.8

By using mrt 8.8.8.8 do not show anything.

Routing table and ARP table

1. Before starting core network

UE

gNB

I-UPF

UPF-PSA1

UPF-PSA2

Core Network

MEC

2.After starting the system(CN、UPFs、gNB)

gNB

• ARP: AMF N2

I-UPF

• Routing: UE
• ARP: SMF N4

UPF-PSA1

• Routing: UE
• ARP: SMF N4

UPF-PSA2

• Routing: UE
• ARP: SMF N4

Core Network

• ARP: gNB N2、I-UPF N4、PSA-UPF1 N4、PSA-UPF2 N4

3. Starting UE2 (Success)

UE2 can access Internet via core network successfully:

UE

• Setting default gatway: 60.61.0.1
• ARP: gNB Nn

gNB

• ARP: UPF-PSA1 N3、UE Nn

PSA-UPF1

• ARP: gNB N3

4. Starting UE1 (Faild)

First, I just start UE1 and set default gateway.

UE

• Setting default gatway: 60.60.0.1

gNB

• ARP: I-UPFN3、UE Nn

I-UPF

• ARP: UPF-PSA2 N9、gNB N3、SMF N4

UPF-PSA2

• ARP: SMF N4、I-UPF N9、I-UPF N3

Trouble shooting

1. Open ARP proxy in gNB and I-UPF
2. Add static route in PSA-UPF2: $sudo route add -net 192.168.0.0/24 dev enp0s9
3. Add static route in I-UPF: $sudo route add -net 192.168.0.0/24 dev enp0s3
4. Add static route in gNB: $sudo route add -net 172.16.9.0/24 dev enp0s8