# xApp Development: OAI and FlexRIC with Free5GC

[TOC]

---

## Architecture


## Pre-Requisites

- **VM 1 [192.168.0.40]** ([OAI Installation](https://hackmd.io/@nomaden/BkFteqYOlx))
- **VM 2 [192.168.0.69]** ([FlexRIC Installation](https://hackmd.io/@nomaden/rJPhM5FOgg))
- **VM 3 [192.168.0.41]** ([Free5GC](https://hackmd.io/@RaffieWinata/SyrHDsL1C#1-Install-Free5GC)) <p style="display: inline; font-size: 12px;">*Make sure to install all the network functions and also the web console module.</p>

## Configuration

### FlexRIC and OAI

You need to copy the ELF (.so) files in FlexRIC VM in `/usr/local/lib/flexric/` to the same path in the OAI VM.

You can either install and build FlexRIC in the OAI VM or just copy and transfer the files using SCP. Example:

# OAI VM
scp -r ubuntu@192.168.0.69:/usr/local/lib/flexric/ /user/local/lib/

### Free5GC

For Free5GC, we don't really need to reconfigure anything. We just need to take some notes of the Network Function configurations as a reference for the OAI configuration and add subcriber for UE.

#### amf

cd ~/free5gc/config/
cat amfcfg.yaml

# ...rest of code
plmnSupportList:
  - plmnId: 
    mcc: 208 # Take note of this
    mnc: 93 # Take note of this
  snssaiList: 
    - sst: 1 # Take note of this
      sd: 010203 # Take note of this
    - sst: 1 
      sd: 112233 
      
supportDnnList:
  - internet # Take note of this
# ...rest of code

#### IP Tables Rules

sudo sysctl -w net.ipv4.ip_forward=1
sudo iptables -t nat -A POSTROUTING -o <dn_interface> -j MASQUERADE
sudo iptables -A FORWARD -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1400

>[!Note] dn_interface
>dn_interface should follow your VM interface, the example are 'enp0s3', 'enp0s8', etc. You can see the name by executing `ip a` command.

#### Add Subscriber (UE) Information

1. Start the Core Network

cd ~/free5gc/
./force_kill.sh # If ran previously
./run.sh # run the Core Network

2. Start the web console

cd webconsole/
go run server.go # run the web UI

3. Go to http://192.168.0.41:5000


**username: admin**
**password: free5gc**

4. Go to Subscribers menu


5. Take information


- Take notes of the IMSI number
- Make sure PLMN ID is the combination of \<MCC\>\<MNC\> from amf configuration
- Take notes of the Operator Code Value
- Take notes of the Permanent Authentication Key (KI)

6. S-NSSAI Configuration
- Delete the second S-NSSAI (01112233)
- Take notes of SST and SD from the first S-NSSAI
- Delete Flow Rules 1 in DNN Configuration

7. Create the Subscribers

### OAI

We'll simulate the O-RAN by splitting the CU and DU of gNB.

#### Openair

In the current OAI version, there is some error with the UE regarding PDU Session Establishment. In order to fix it, we need to modify the source code and rebuild the OAI. The issue was mentioned here: [Link](https://gitlab.eurecom.fr/oai/openairinterface5g/-/issues/931).

cd ~/oai/openair3/NAS
sudo nano ./NR_UE/5GS/5GSM/MSG/PduSessionEstablishmentAccept.c

:::spoiler Code
:::

sudo nano ./NR_UE/nr_nas_msg.c

:::spoiler Code
:::

#### CU

Open file

sudo nano ~/oai/targets/PROJECTS/GENERIC-NR-5GC/CONF/cu_gnb.conf

:::spoiler Change configurations

# ... rest of code
gNB_name  =  "gNB-Eurecom-CU";

tracking_area_code  =  1;
plmn_list = ({ 
    mcc = 208; # Make sure match with amf configurations 
    mnc = 93; # Make sure match with amf configurations 
    mnc_length = 2; 
    snssaiList = ({ 
        sst = 1; # Make sure match with subscriber
        sd = 0x010203 # Make sure match with subscriber
      }) 
    });
    
# ... 

////////// AMF parameters:
# make sure match with Free5GC VM IP address
amf_ip_address = ({ ipv4 = "192.168.0.41"; }); 

NETWORK_INTERFACES :
{
    # make sure match with OAI VM IP address
    GNB_IPV4_ADDRESS_FOR_NG_AMF              = "192.168.0.40";
    # make sure match with OAI VM IP address
    GNB_IPV4_ADDRESS_FOR_NGU                 = "192.168.0.40";
    GNB_PORT_FOR_S1U                         = 2152; # Spec 2152
};
    
# ... end of code
# insert below code
    
e2_agent = {
  # make sure match with FlexRIC VM IP address
  near_ric_ip_addr = "192.168.0.69";
  sm_dir = "/usr/local/lib/flexric/";
};

:::

#### DU

Open file

sudo nano ~/oai/targets/PROJECTS/GENERIC-NR-5GC/CONF/du_gnb.conf

:::spoiler Change configurations

# ... rest of code
gNB_name  =  "gNB-Eurecom-CU";

tracking_area_code  =  1;
plmn_list = ({ 
    mcc = 208; # Make sure match with amf configurations 
    mnc = 93; # Make sure match with amf configurations 
    mnc_length = 2; 
    snssaiList = ({ 
        sst = 1; # Make sure match with subscriber
        sd = 0x010203 # Make sure match with subscriber
      }) 
    });

# ... end of code
# insert below code
    
e2_agent = {
  # make sure match with FlexRIC VM IP address
  near_ric_ip_addr = "192.168.0.69";
  sm_dir = "/usr/local/lib/flexric/";
};

:::

#### UE

Open file

sudo nano ~/oai/targets/PROJECTS/GENERIC-NR-5GC/CONF/ue.conf

:::spoiler Change Configurations

# Make sure to match all parameters to be the same as subscriber information
uicc0 = {
imsi = "208930000000001"; 
key = "8baf473f2f8fd09487cccbd7097c6862";
opc= "8e27b6af0e692e750f32667a3b14605d";
dnn= "internet";
plmn = "20893";
nssai_sst=1;
nssai_sd=0x010203;
}

# We dont really need to determine the UE position
#position0 = {
#    x = 0.0;
#    y = 0.0;
#    z = 6377900.0;
#}

# @include "channelmod_rfsimu_LEO_satellite.conf"

:::

### FlexRIC

We need to change the RIC IP Address to match with the VM IP and not using loopback (127.0.0.1).

Open file

sudo nano /usr/local/etc/flexric/flexric.conf

:::spoiler Change Configurations

[NEAR-RIC]
NEAR_RIC_IP = 192.168.0.69

[XAPP]
DB_DIR = /tmp/

:::

<br >

Or you can just change the default configuration in `~/flexric/flexric.conf`.

## Change Source Code

### OAI

For reading KPM, when filling the UE related information, we need to make sure the UE information list is available.

**`openair2/E2AP/RAN_FUNCTION/O-RAN/ran_func_kpm.c`:**

:::spoiler Code

static cudu_ue_info_pair_t fill_ue_related_info(arr_ue_id_t* arr_ue_id, const size_t ue_idx)
{
  cudu_ue_info_pair_t ue_info = {0};

  if (arr_ue_id->ue_id[ue_idx].type == GNB_UE_ID_E2SM) {
    ue_info.rrc_ue_id = *arr_ue_id->ue_id[ue_idx].gnb.ran_ue_id;  // rrc_ue_id
    if (arr_ue_id->ue_info_list)      // <-- guard
      ue_info.ue = arr_ue_id->ue_info_list[ue_idx];
  } else if (arr_ue_id->ue_id[ue_idx].type == GNB_CU_UP_UE_ID_E2SM) {
    /* in OAI implementation, CU-UP ue id = CU-CP ue id
                           => CU-UP ue id = rrc_ue_id, but it should not be the case by the spec */
    ue_info.rrc_ue_id = *arr_ue_id->ue_id[ue_idx].gnb_cu_up.ran_ue_id;  // cucp_ue_id = rrc_ue_id
  } else if (arr_ue_id->ue_id[ue_idx].type == GNB_DU_UE_ID_E2SM) {
    ue_info.rrc_ue_id = *arr_ue_id->ue_id[ue_idx].gnb_du.ran_ue_id;  // rrc_ue_id
    if (arr_ue_id->ue_info_list)
      ue_info.ue = arr_ue_id->ue_info_list[ue_idx];
  }
  
  return ue_info;
}

:::

<br >

We also need to change the memory allocation size to allocate `sd` resource:

:::spoiler Code

static void capture_sst_sd(test_cond_value_t* test_cond_value, uint8_t *sst, uint32_t **sd)
{
  DevAssert(sst != NULL);
  DevAssert(sd != NULL);

  // S-NSSAI is an OCTET_STRING, as defined by spec
  switch (test_cond_value->type) {
    case OCTET_STRING_TEST_COND_VALUE: {
      if (test_cond_value->octet_string_value->len == 1) {
        *sst = test_cond_value->octet_string_value->buf[0];
        *sd = NULL;
      } else {
        DevAssert(test_cond_value->octet_string_value->len == 4);
        uint8_t *buf = test_cond_value->octet_string_value->buf;
        *sst = buf[0];
        *sd = malloc(sizeof(uint32_t));
        **sd = buf[1] << 16 | buf[2] << 8 | buf[3];
      }
      break;
    }
    default:
      AssertFatal(false, "test condition value %d impossible\n", test_cond_value->type);
  }
}

:::

<br >

While for the RC, we need to comment some assertions since its not necessary for this case.

**`openair2/E2AP/RAN_FUNCTION/O-RAN/ran_func_rc.c`:**

:::spoiler Code

sm_ag_if_ans_t write_ctrl_rc_sm(void const* data)
{
  assert(data != NULL);
//  assert(data->type == RAN_CONTROL_CTRL_V1_03 );

  rc_ctrl_req_data_t const* ctrl = (rc_ctrl_req_data_t const*)data;

  assert(ctrl->hdr.format == FORMAT_1_E2SM_RC_CTRL_HDR && "Indication Header Format received not valid");
  assert(ctrl->msg.format == FORMAT_1_E2SM_RC_CTRL_MSG && "Indication Message Format received not valid");
  assert(ctrl->hdr.frmt_1.ctrl_act_id == 2 && "Currently only QoS flow mapping configuration supported");

  printf("QoS flow mapping configuration\n");

  const seq_ran_param_t* ran_param = ctrl->msg.frmt_1.ran_param;

  // DRB ID
  assert(ran_param[0].ran_param_id == 1 && "First RAN Parameter ID has to be DRB ID");
  // assert(ran_param[0].ran_param_val.type == ELEMENT_KEY_FLAG_TRUE_RAN_PARAMETER_VAL_TYPE);
  printf("DRB ID %ld \n", ran_param[0].ran_param_val.flag_true->int_ran);


  // List of QoS Flows to be modified in DRB
  assert(ran_param[1].ran_param_id == 2 && "Second RAN Parameter ID has to be List of QoS Flows");
  // assert(ran_param[1].ran_param_val.type == LIST_RAN_PARAMETER_VAL_TYPE);
  printf("List of QoS Flows to be modified in DRB\n");
  const lst_ran_param_t* lrp = ran_param[1].ran_param_val.lst->lst_ran_param;

  // The following assertion should be true, but there is a bug in the std
  // check src/sm/rc_sm/enc/rc_enc_asn.c:1085 and src/sm/rc_sm/enc/rc_enc_asn.c:984 
  // assert(lrp->ran_param_struct.ran_param_struct[0].ran_param_id == 3);

  // QoS Flow Identifier
  assert(lrp->ran_param_struct.ran_param_struct[0].ran_param_id == 4);
  // assert(lrp->ran_param_struct.ran_param_struct[0].ran_param_val.type == ELEMENT_KEY_FLAG_TRUE_RAN_PARAMETER_VAL_TYPE);
  int64_t qfi = lrp->ran_param_struct.ran_param_struct[0].ran_param_val.flag_true->int_ran;
  assert(qfi > -1 && qfi < 65);

  // QoS Flow Mapping Indication
  assert(lrp->ran_param_struct.ran_param_struct[1].ran_param_id == 5);
  // assert(lrp->ran_param_struct.ran_param_struct[1].ran_param_val.type == ELEMENT_KEY_FLAG_FALSE_RAN_PARAMETER_VAL_TYPE);
  int64_t dir = lrp->ran_param_struct.ran_param_struct[1].ran_param_val.flag_false->int_ran;
  assert(dir == 0 || dir == 1);

  printf("qfi = %ld, dir %ld \n", qfi, dir);


  sm_ag_if_ans_t ans = {.type = CTRL_OUTCOME_SM_AG_IF_ANS_V0};
  ans.ctrl_out.type = RAN_CTRL_V1_3_AGENT_IF_CTRL_ANS_V0;
  return ans;
}

:::

## Running the Environment

### Core Network

- Make sure the Core Network is running

    cd ~/free5gc/
    ./run.sh

:::spoiler Result
:::
- [Optional] Run the web console
:::spoiler Result
:::

### FlexRIC

cd ~/flexric/build/examples/ric
./nearRT-RIC

:::spoiler Result
:::

### O-RAN

#### CU

cd ~/oai/cmake_targets/ran_build/build/
sudo ./nr-softmodem -O  ~/oai/targets/PROJECTS/GENERIC-NR-5GC/CONF/cu_gnb.conf --rfsim --sa -E

:::spoiler Result
:::

#### DU

cd ~/oai/cmake_targets/ran_build/build/
sudo ./nr-softmodem -O ~/oai/targets/PROJECTS/GENERIC-NR-5GC/CONF/du_gnb.conf --rfsim --sa -E

:::spoiler Result
:::

#### UE

cd ~/oai/cmake_targets/ran_build/build/
sudo ./nr-uesoftmodem -r 106 --numerology 1 --band 78 -C 3619200000 --ssb 516 --rfsim -E -O ~/oai/targets/PROJECTS/GENERIC-NR-5GC/CONF/ue.conf --rfsimulator.serveraddr 192.168.0.40

:::spoiler Result
:::

## Testing

The scheme for this testing is that the UE (subscriber) doesn't have Flow Rules in the DNN Configurations *(Its deleted in the previous section)*.

### Core Network

Initial Sync

:::spoiler Result
:::

RRC and NAS

:::spoiler Result

:::

### Near RT RIC

Running the RIC

:::spoiler Result
:::

Succesfully Communicate with E2 Agent

:::spoiler Result
:::

### Result

CU can receive the message:

:::spoiler Result for KPM
:::

:::spoiler Result for RC Control
:::

<br >

However, the E2 Agent couldn't find the UE that matched the condition given by xApp. If we take a look at `examples/xApp/c/kpm_rc/xapp_kpm_rc.c`:

:::spoiler Code

static
kpm_act_def_t fill_report_style_4(ric_report_style_item_t const* report_item)
{
  assert(report_item != NULL);
  assert(report_item->act_def_format_type == FORMAT_4_ACTION_DEFINITION);

  kpm_act_def_t act_def = {.type = FORMAT_4_ACTION_DEFINITION};

  // Fill matching condition
  // [1, 32768]
  act_def.frm_4.matching_cond_lst_len = 1;
  act_def.frm_4.matching_cond_lst = calloc(act_def.frm_4.matching_cond_lst_len, sizeof(matching_condition_format_4_lst_t));
  assert(act_def.frm_4.matching_cond_lst != NULL && "Memory exhausted");
  // Filter connected UEs by S-NSSAI criteria
  test_cond_type_e const type = S_NSSAI_TEST_COND_TYPE; // CQI_TEST_COND_TYPE
  test_cond_e const condition = EQUAL_TEST_COND; // GREATERTHAN_TEST_COND
  int const value = 1;
  act_def.frm_4.matching_cond_lst[0].test_info_lst = filter_predicate(type, condition, value);

  // Fill Action Definition Format 1
  // 8.2.1.2.1
  act_def.frm_4.action_def_format_1 = fill_act_def_frm_1(report_item);

  return act_def;
}

:::

We can see that the xApp condition is to match the S-NSSAI and the xApp fill the value to be 1. However we need to match with `SST` and also the `SD` that we setup in the **Core Network**. So we need to change the condition:

:::spoiler Code

static
kpm_act_def_t fill_report_style_4(ric_report_style_item_t const* report_item) {
  assert(report_item != NULL);
  assert(report_item->act_def_format_type == FORMAT_4_ACTION_DEFINITION);

  kpm_act_def_t act_def = { .type = FORMAT_4_ACTION_DEFINITION };

  // Fill matching condition
  // [1, 32768]
  act_def.frm_4.matching_cond_lst_len = 1;
  act_def.frm_4.matching_cond_lst = calloc(act_def.frm_4.matching_cond_lst_len, sizeof(matching_condition_format_4_lst_t));
  assert(act_def.frm_4.matching_cond_lst != NULL && "Memory exhausted");
  // Filter connected UEs by S-NSSAI criteria
  test_cond_type_e const type = S_NSSAI_TEST_COND_TYPE; // CQI_TEST_COND_TYPE
  test_cond_e const condition = EQUAL_TEST_COND; // GREATERTHAN_TEST_COND

  int const value = 1;
  uint8_t snssai_bytes[4] = { 0x01, 0x01, 0x02, 0x03 }; // SST=1, SD=010203

  test_info_lst_t snssai_cond = { 0 };
  snssai_cond.test_cond_type = S_NSSAI_TEST_COND_TYPE;
  snssai_cond.test_cond = calloc(1, sizeof(test_cond_e));
  *snssai_cond.test_cond = EQUAL_TEST_COND;

  // allocate bit string value
  snssai_cond.test_cond_value = calloc(1, sizeof(test_cond_value_t));
  snssai_cond.test_cond_value->type = OCTET_STRING_TEST_COND_VALUE;
  snssai_cond.test_cond_value->octet_string_value = calloc(1, sizeof(byte_array_t));
  snssai_cond.test_cond_value->octet_string_value->len = sizeof(snssai_bytes);
  snssai_cond.test_cond_value->octet_string_value->buf = calloc(sizeof(snssai_bytes), 1);
  memcpy(snssai_cond.test_cond_value->octet_string_value->buf, snssai_bytes, sizeof(snssai_bytes));

  act_def.frm_4.matching_cond_lst[0].test_info_lst = snssai_cond;

  // Fill Action Definition Format 1
  // 8.2.1.2.1
  act_def.frm_4.action_def_format_1 = fill_act_def_frm_1(report_item);

  return act_def;
}

:::

After changing the code, build the FlexRIC again and run again:

**CU**:

:::spoiler Result
:::

**xApp**:

:::spoiler Result
:::

As you can see, the KPM information is displayed in the xApp. And if we try to increase data transfer by using `iperf3`:

- On OAI VM:
`iperf3 -s -B <UE IP Address>` (from oaitun_ue1)
- On Core Network VM:
` iperf3 -c <UE IP Address>`

We can see the result in bandwidth increase from KPM xApp:

:::spoiler Result
:::

## Limitations

For the current OAI and FlexRIC implementation, If we take a look at the OAI source code:

:::spoiler Code
:::

You could see that the condition only supports EQUAL condition, meaning we can only target one service.

---

The other is the RC Control right now only support QoS Flow Mapping configuration.

:::spoiler Code
:::

---

And the last one is only my deduction, I don't know if this is correct or not. But after sending the RC Control with the new qfi number. The E2 Agent only acknowledge the Control message but not triggering the remapping of the QoS configuration.

:::spoiler Code
:::