Internship Notes by Kevin
===
###### tags: `Internship`

_________________________

##### [<center>Root Page</center>](https://hackmd.io/i5BFIooYTKyT7B1JUfAznA?view)
>Name : Kevin Darmawan
>Departement : Dept. of Electrical Engineering (Computer engineering)
>University : Universitas Indonesia
>General Topic : Front-haul Interface
_________________________

### Week 01 : Background of O-RAN

- Fundamentals: Introduction to Open RAN ([2021/02/02](https://hackmd.io/PTzHdx4uRKeGq8nDXznyyw))
- Fundamentals: Introduction to 5G ([2020/02/03](https://hackmd.io/G0CUmDToQy6ruuhbTrBykA))
- Study Review : Overview of Fronthaul Interface ([2020/02/05](https://hackmd.io/QHRLYAtaS0aWOyDz1glyEA?edit))

:::success
**Goals**
- [x] Learn about the background knowledge of O-RAN, how the system works and identify parts of the general architecture.
- [x] Understand the what is 5G, know about mobile network evolution until 5G, how it works, learn about the 5G New Radio technology and explore what is beyond 5G
- [x] Explore a topic of interest : Fronhaul Interface

\
**What I have learned**
- An understanding of what is 5G mobile technology and the improvements over the past generation
- A quick introduction to O-RAN, the why, what, who, when, where, how
- Brief overview about FHI, why it is important to O-RAN
:::

### Week 2 : OSC Fronthaul Library
- Exploring OSC documentation and clone repo ([2021/02/09](https://hackmd.io/NCV02-jGSEiUvI8qHsX81g))
- OSC Fronthaul Library
- [Part 1](https://hackmd.io/fRgXCg5pQZeNNqZVwQJo9g)
- [Part 2](https://hackmd.io/vKbT_NpcT8OgBLtVSTuG-w)
- [Part 3](https://hackmd.io/bc5WfTFmTcOKnLlYXDYC3g)
- OSC vs OAI Fronthaul Library ([20219/02/12](https://hackmd.io/Uvo5NcSASlmEMVVMIfrjmA))

:::success
**Goals**
- [x] Understand what is the fronthaul library, what does it do, how to use the library
- [x] As an open source, explore different projects developing the fronthaul library
- [x] Develop an understanding on how to implement code (FHlib) to the architecture of Fronthaul interface

\
**What I have learned**
- The fronthaul library is based on xran library built using DPDK
- Different projects in O-RAN FH means we need to work on integration with other projects
- There is also a useful sample-app that can be used to understand how the FH uses xran library
:::

### Week 3 : Fronthaul Transport System
- Reviewing back on wireless communication ([2021/02/15](https://hackmd.io/Tdot7CArQo-HtuC5iLhoTQ))
- Understanding the 5G NR Physical Layer ([2021/02/16](https://hackmd.io/_S8KxkXWTrujkWHO3xpJDg))
:::warning
At this point of the pre-internship is to finish the task given by the mentor to quickly get around the spesifications of FH and implement it later on. So, topic of pre-internship notes may be on hold and replaced by necessary task related studies anytime.
:::
- Task update : Mapping the FH OSC source code ([2021/02/17](https://hackmd.io/vT11T2UeSxiKOjJhGcylzQ))

### Week 4 : Functionality of the C-Plane and U-Plane
- [O-RAN Fronthaul spesification overview](https://hackmd.io/FdZs5MLzStCYQ0FXryDzLA) (2021/02/19)
- [C-Plane Section](https://hackmd.io/vlwAEidOS2C8-ydUNNh21A) (2021/02/22)
- [part 2](https://hackmd.io/-QwJc4iUSDSpXS_RJELb9A) (2021/02/23)
- [U-Plane Section](https://hackmd.io/nC2umNcjQMW9lqOYOsFoFA) (2021/02/25)

:::success
**Goals**
- [x] Understand what functions are done in the C-Plane and U-Plane
- [x] Return a mapping figure of what functions of the fronthaul are specified by O-RAN in comparison to what functions are available on the OSC source code and Intel FPGA

\
**What I have learned**
- Functionalities that are required in O-RAN FH are stated within the spec, resume in the weekly notes
- In comparison to the spesifications, development of the OSC source codes are not finished espescially at the time for the C-Plane, the functions currently supported are scheduling and precoding. U-Plane functions have been implemented.
:::

### Week 5 : Mapping figures for C-Plane and U-Plane messages
Tables of code and spec comparison are updated here:
- [C-Plane messages](https://hackmd.io/vlwAEidOS2C8-ydUNNh21A) (2021/03/01)
- [U-Plane messages](https://hackmd.io/nC2umNcjQMW9lqOYOsFoFA) (2021/03/02)
- [Data Transfer prequisites (delay management)](https://hackmd.io/KlHU4GASRp6bdyfkgayoiw) (2021/03/05)

:::success
**Goals**
- [x] Understand the differences of C-Plane messages and how they are used to carry out different functions
- [x] Understand the differences of U-Plane messages and how they are used to carry out different functions
- [x] Return mapping figures between the O-RAN FH spec and OSC source codes

\
**What I have learned**
- Messages have header fields that identify their functions and their content, some use common header and some used for spesific function
- Control plane messages define the scheduling, coordination required for data transfer,  beam-forming etc.
- User plane messages for efficient data transfer within the strict time limits of 5G numerologies.
:::

### Week 6 : Reading codes
- [Tracing packets](https://hackmd.io/Mpzqnjn7TfOkXDdspoMfJw) (2021/03/08)

:::success
**Goals**
- [x] Previous documentations on FH spec and code figures need restructuring for easier reading
- [x] Understand how the C-Plane and U-Plane is initiated
- [x] Dive into the source code to better understand process that is happening in the documentations

\
**What I have learned**
- Understanding the concept, process, and spesification of FH is thoroughly needed for implementation, various study sources and better method for code reading must be collected to better understand the source code
- As the OSC FHlib is still under progress and some functions haven't been completed, some functions of the sample app are done by directly accessing the xran library.
- So far the packet buffer definitions have made use of the FH lib.
:::

### Week 7 : C-Plane function in code
- [C-Plane Scheduling and Beamforming Commands](https://hackmd.io/kiBpqbyYQTOgomi9jvPa6A) (2021/03/15)
- [Read OSC code with Doxygen](https://hackmd.io/Mq9aUG8JQ5aVLjRIxzmYyw) (2021/13/17)
:::success
**Goals**
- [X] Understand process of Scheduling and Beamforming commands of the C-Plane
- [X] Understand and map functions related to Mixed Numerologies and PRACH handling of C-Plane
- Understand process of IQ data transfer procedure :arrow_left: *moved to next weeks*

\
**What I have learned**
- Discovered the OSC documentation created using **Doxygen** which helps for searching variables / structs / functions throughout the entire repository, locating it faster, but how the html documentation will look like on different systems may differ.
- Understood the structure of C-Plane messages using eCPRI format (headers and section) and relation to the code
- Understood what mixed numerologies in O-RAN FH mean and that *it affects the periodicity of C-Plane messages sent* (~slot) and map which have been implemented in table, although understanding how to use functions of the parameters require more reading.
:::

### Week 8 : U-Plane function in code
- [U-Plane Functions - HackMD](https://hackmd.io/sexllI4DTs2kyxz6qwAvkA) (2021/03/23)
:::success
**Goals**
- [X] Understand process of IQ data transfer procedure
- Map the process and or functions of the code in figures and or tables

\
**What I have learned**
- Understood the main functions of U-Plane and those functions that are implemented in OSC. U-Plane messages are sent in a symbol by symbol manner following the control messages sent before, delay management parameters are also different for DL/UL
- The process of mapping the whole procedure seems a little too complicated for now, because tracing the sample app, more functions of the xran library is called than the OSC FH library.
:::

### Week 9 : OSC FH Library Structure
- [OSC FH library structure - HackMD](https://hackmd.io/LGYWfqOqTDutbGP9shOuRw) (2021/03/31)

:::success
**Goals**
- [X] Create figures that describe the hierarchy and dependency of OSC FH library
- Explain the code functions related to C-Plane and U-Plane

\
**What I have learned**
- OSC FH library utilizes XRAN library functions using APIs for the Fronthaul functions, these functions are organized by **xran_fh_o_du.h** according to split 7.2x
- Functions related to U-Plane API are related to IQ symbol packing and unpacking, encapsulation with U-Plane headers and ethernet headers. Compression methods e.g BFP algorithms use an external API. C-Plane on progress
:::

### Week 9 & 10 : Midterm test and ramadan work hour adjustments

### Week 11 : FHI Initialization and Configuration
:::info
**Goals**
- [x] Understand how the configuration file for FHI works and what parameters are neede
- [x] Understand and create diagram on how Fronthaul initialization is done.
:::
---
## Summer Intern Starting :sun_with_face:
### Week 12 :
- [Fronthaul Interface Ethernet Data Layer](https://hackmd.io/ZLFxBBj8QMuzcfFR6aPDoQ)
- [Fronthaul Interface Ethernet for Transport](https://hackmd.io/WCWUmJAsRJG4qBQKFLYmNg) (2021/07/02)
- [O-RU entity of Fronthaul Interface](https://hackmd.io/4DHXWtEjSv-su7nQR7vaWw) (2021/07/05)
- [Unit Testing on FHlib wrapper](https://hackmd.io/_WJu5SY9S3ar0VgWqWY6Qg) (2021/07/06)

:::success
**What I have learned**
- Ethernet driver modules `phy/fhi_lib/lib/ethernet` in OSC uses ethdi.c to call DPDK and inititate data transfer over the Ethernet
- To verify that all functions of the FHI is running and ready to be deployed we can use the unittest provided by OSC
- Understand that functions are implemented through a wrapper code
:::

### Week 13 :
- [Testing build FHlib](https://hackmd.io/YWvtEzOYSgyILqRla9M2pQ) (2021/07/07)
- [Wrapper codes and debugging](https://hackmd.io/WruFR5agRhKsFrmzTqtVVA) (2021/07/08)
- [Makefiles in FHlib](https://hackmd.io/VJ-qTRexSHW8yuHKUDIc7A) (2021/07/09)
- [General Call Flow in OSC](https://hackmd.io/RxpZ_ADNTXevHYw1Dqqxsw) (2021/07/12)
- [DPDK Functions used in transfer/recieve](/WWanlmoBRLyrfVz9A1xGAw) (2021/07/13)

:::success
**Goals**
- [X] Learn how to do unit testing for OSC Fronthaul Interface
- [X] Understand the code flow in general
- [X] Identify where in FHIlibrary, DPDK functions are used
---
**What I have learned**
- Unit tests for the FHI verify if the installation and configuration of FHI library is correct and functions are working properly. Factors as experience with server, hardware capabilities, understanding configuration is important.
- Created figure for general call flow [link](https://hackmd.io/RxpZ_ADNTXevHYw1Dqqxsw). More appropriate and detailed references can be found in [FHI project page Task 1.6](https://hackmd.io/l8qTQYTeRsmncjx8M1miEg?view#Fronthaul-Interface-FHI)
- DPDK functions that can be found in process of C-plane and U-plane TX/RX are (rte_ring, rte_pktmbuf, Ethernet Poll Mode Driver). Other core functions are used in FH but not directly involved.
- The rte_ring lockless Ring buffer is used to load packets from the memory buffer to manage them properly for packet processing and TX/RX.
:::

### Week 14 :

- [DPDK Packet Buffers](/l_BvxfiJSD2lMY4AQNLvbQ) (2021/07/14)
- [Status for O-DU integration](/vsNgJig_TpC4DMSmN-1DSw) (2021/07/15)
- [PTP synchronization](/DC6FfyTCQ6SKO7tnOvyQlg) (2021/07/16)
- [DPDK Ethernet Poll Mode Driver](/_mRya0r0RFCSrbiudUYLWQ) (2021/07/19)

:::success
**Goals**
- [X] Follow up on the current status of PHY code
- It's already fixed in bronze release, nothing changed
- [X] Learn about PTP synchronization
- [X] Figure out how DPDK functions are used in FHI data transfer

---

**What I have learned**
- PTP synchronization in the lab for LLS-C3 cannot be finished because there is no Grandmaster Clock. For testing, LLS-C1 where O-DU sync O-RU directly can be used (*need to read how to implement*)
- From the previous notes about DPDK we can say that the FHI creates 3 threads:
- For managing packet timing/delay for RX/TX `xran_timing_source_thread()` [1](https://hackmd.io/l_BvxfiJSD2lMY4AQNLvbQ) ; [2](https://hackmd.io/8t2AhM-6QjWSy9j3l91owA)
- For running Ethernet Poll Mode Drivers `process_dpdk_io()` [note :book:](https://hackmd.io/_mRya0r0RFCSrbiudUYLWQ)
- For processing recieved packets in the packet queues `ring_processing_thread()` [note :book:](https://hackmd.io/WWanlmoBRLyrfVz9A1xGAw)
:::spoiler DPDK threads OSC FH
:::

### Week 15 :
- [DPDK Shared Memory](/b3j9xO85R3e0tPYdq0S8mw) (2021/07/21)
- [PTP Synchronization standards](https://hackmd.io/RYOqjzGzTaO5TJV87UoelQ) (2021/07/22)
- [Timing and synchronization API](/AaiIsD4TSiqOtKRtE-ZxmQ) (2021/07/23)
- [DPDK timers](/tqaiB4-DRym-ugsuC8snnQ) (2021/07/26)
- [DPDK functions summary](/ij_Zvjm_R9u_-uuc5swJgQ) (2021/07/27)

:::success
**Goals**
- [x] Complete and recap about DPDK functions
- [x] Investigate the implementation of S-Plane protocols

---

**What I have learned**
- PTP synchronization standards IEEE 1588 is fulfilled using LinuxPTP. Information about ptp4l can be fetched through pmc
- [pmc issue patch](https://hackmd.io/RYOqjzGzTaO5TJV87UoelQ#PTP-Management-Client-PMC-utility)
- XRAN does not have a synchronization API on it's own, it only checks based on linuxPTP. The timing API for timing and delay management is also obtained from system time synced with PTP.
- [link to note](/AaiIsD4TSiqOtKRtE-ZxmQ)
- Summary of DPDK functions
- [link to note](https://hackmd.io/ij_Zvjm_R9u_-uuc5swJgQ)

:::

### Week 16 :
- [PMC troubleshooting](/wKJ6AV4jRieQwHrHPMKUdQ) (2021/07/28)
- [Applying pmc fix](/wKJ6AV4jRieQwHrHPMKUdQ?both#A-Error-command-not-found-2907) (2021/07/29)
- [PTP Implementation Study](/mI-3obN2Siil6By6up88ZQ) (2021/07/30)
- [PMC troubleshooting | Issue #2 pmc: command not found](https://hackmd.io/wKJ6AV4jRieQwHrHPMKUdQ?view#B-Issue-2-Fix) (2021/02/08)
- [Troubleshooting PTP Timestamps ](/JV-9OAsOTpG0yZUceeGr1A) (2021/08/03)

:::success
**Goals:**
- [X] Troubleshoot pmc issues [FHI Discussion - HackMD](https://hackmd.io/903nREczSNmNhjH6IyTVHQ#715-prof-Ray--Send-to-Alex)
- [X] Design PTP topology for End-to-End simulation
- Attempt troubleshoot PTP timestamp issues

---

**What I have learned:**
- pmc function is to gather PTP data from all nodes, used for knowing port state
- Successfully solved the PTP issue
- ["Failed to create pmc"](https://hackmd.io/wKJ6AV4jRieQwHrHPMKUdQ?both#A-Issue-1-fix)
- ["sudo: pmc: command not found"](https://hackmd.io/wKJ6AV4jRieQwHrHPMKUdQ?both#B-Issue-2-Fix)
- I do not consider "PTP recieved without timestamp" (*until now*) as a hardware issue, the fix is published this week. Will need to verify it or install a newer driver
- Proposed a topology for the current O-DU and O-RU simulator in the lab for direct synchronization with linuxptp [PTP Implementation Study - HackMD](https://hackmd.io/mI-3obN2Siil6By6up88ZQ)

:::

### Week 17:
- [Troubleshooting PTP Timestamps](https://hackmd.io/JV-9OAsOTpG0yZUceeGr1A?view#A-Configure-hwstamp_ctl) (2021/08/04)
- [Troubleshooting PTP Timestamps](https://hackmd.io/JV-9OAsOTpG0yZUceeGr1A#Update-08052021) (2021/08/05)
- [O-RAN Management Plane - HackMD](https://hackmd.io/6lxOmH6zTweH-TeqHQHucw) (2021/08/06)
- [NETCONF Protocol](/NfHix19jRxC4lgO2vAWcdQ) (2021/08/09)
- [ M-Plane O-DU and O-RU interface management ](/ndEjhTSkSvqOguDNrINQbQ) (2021/08/10)

:::success
**Goals:**
- [x] Configure hardware timestamping through hwstamp_ctl utility
- [x] Find out about implementation of M-Plane in Fronthaul interface

---

**What I have learned:**
- Certain versions of NIC firmware allow users to change hardware timestamping parameters without having to change the driver using `hwstamp_ctl`. *Unfortunately this did not work*.
- More about the theory and results [in this section](https://hackmd.io/JV-9OAsOTpG0yZUceeGr1A?view#A-Configure-hwstamp_ctl)
- M-Plane has different data flow than C/U/S plane, it uses NETCONF protocol stack and YANG for management capabilities
- [MP intro](https://hackmd.io/6lxOmH6zTweH-TeqHQHucw)
- [NETCONF Protocol](/NfHix19jRxC4lgO2vAWcdQ)
- [ M-Plane O-DU and O-RU interface management ](/ndEjhTSkSvqOguDNrINQbQ)
:::

### Week 18:
- [Troubleshooting PTP Timestamps - HackMD](https://hackmd.io/JV-9OAsOTpG0yZUceeGr1A?both#B-Test-Using-Software-Timestamping)
(2021/08/12)
- [ PTPv2 Initiation Script](/lqx1wp8JSkadJWmzScggYg) (2021/08/13)
- [Troubleshooting PTP Timestamps - Update drivers/kernels](https://hackmd.io/JV-9OAsOTpG0yZUceeGr1A?both#Update-16082021) (2021/08/16)

:::success
**Goals:**
- [x] Attempt alternative S-Plane solutions
- [x] Analyze other hardware timestamping problem causes

---

**What I have learned**
- Although the accuracy may not meet O-RAN spec standards, using software timestamp may be used in testing environments to check if PTP is working.
- There might be slight differences in the Intel XL710 NIC model of the two servers. We also attempt to update latest drivers
:::

### Week 19:
- [ S-plane for timestamp generation reference ](/JekGo9XEQJGOwk-6EXIIjA) (2021/08/19)