# <center> Troubleshooting PTP Timestamps </center>
###### tags: `O-RAN`
:::info
**Goal:**
The goal is to solve the issue where PTP slave cannot read timestamps from the master.
- [x] Possibility of hardware issues
- [x] Procedure of sync
- [X] Configuration of PTP4l
- [ ] Succeed and get PTP Hardware Timestamps
:::
:::warning
**References:**
- [Precision Time Protocol on Linux ~ Introduction to linuxptp (linuxfound.org)](https://events.static.linuxfound.org/sites/events/files/slides/lcjp14_ichikawa_0.pdf)
- [PTP Basics (nettimelogic.com)](https://www.nettimelogic.com/resources/PTP%20Basics.pdf)
- [hwstamp_ctl(8) \[centos man page\] (unix.com)](https://www.unix.com/man-page/centos/8/HWSTAMP_CTL/)
- [ptp4l(8) \[centos man page\] (unix.com)](https://www.unix.com/man-page/centos/8/ptp4l/)
:::

:::danger
**Issue brief - problem in AEEON ip-120**
The issue is about PTP slave cannot read timestamps of packets. The problem might be a hardware issue, but according to OSC documentation that shouldn't be the main cause because the NIC is capable hardware-wise
\
> Issue thread: [link](https://hackmd.io/903nREczSNmNhjH6IyTVHQ#715-prof-Ray--Send-to-Alex)

:::

:::info
**Temporary solution**
[Use software timestamping instead of hardware (NIC)](https://hackmd.io/JV-9OAsOTpG0yZUceeGr1A?edit)

:::

## I. NIC spesifications
### A. Spesification comparison
| NTUST NIC | Recommended NIC |
| ------------------------------------ | ------------------------------------ |
|  |  |

### B. Kernel version
> Kernel configurations [link](https://hackmd.io/ubZAS99tTIecVQiOTpvKlA)

### C. Comparison Analysis
- **Driver-wise** NTUST's NIC already has i40e driver installed with the i40e drivers. **However**, the version installed (2.8.20-k) **is older** than the OSC recommended version (2.10.19.82). O-RAN OSC also states that:

2.Download from Intel Website and install updated version of i40e driver if needed.
The current recommended version of i40e is 2.10.19.82. 
However, any latest version of i40e after x2.9.21 expected to be functional for ORAN FH.

==Driver version== may be a factor why timestamps are not working (hypothesis).

- **Hardware-wise** the NIC supports the recommended Capabilities and Hardware Transmit Timestamp and Hardware Recieve Filter modes.
- **Kernel-wise** the system uses version 3.10.0 and it should not be a problem as it is the minimal version for [OSC PTP implementation](https://docs.o-ran-sc.org/projects/o-ran-sc-o-du-phy/en/latest/PTP-configuration_fh.html?highlight=3.10#ptp-for-linux-requirements)

:::info
**Verdict** : Hardware compatibility should not be a problem, but driver may be the source of problem
:::
## II. Procedure of PTP Sync
### A. Theory
Referenced from [this document](https://events.static.linuxfound.org/sites/events/files/slides/lcjp14_ichikawa_0.pdf), Here is how packet timestamping works in Linux PTP.
**Procedure:**
1. Master sends `sync` and optional `follow-up` message
- timestamps taken on both sides t1 & t2
2. After recieve `sync`, slave sends `delay_req` and dakes timestamp t3 on transmit
3. Master recieves `delay_req` and takes timestamp t4
4. Master sends `delay_resp` containing t4 timestamp (time of `delay_req` recieved)
5. Slave recieves `delay_resp` message
- no timestamps taken
- slave calculates End-to-End offset

:::info
**Information**: The 'sync' and 'delay_req' must be timestamped at both sides when transmit/recieving to get t1-t4 and calculate offset. Some timestamps missing may cause PTP to be stuck at UNCALIBRATED
:::

### B. Example Results
If we successfully configure PTP timestamping then the output should look like on the following slave:

- PTP4l uses PTPv2 standard implementation according to IEEE 1588 (not backwards compatible)
- The ptp4l command uses default_slave.cfg provided by OSC, nothing too special here
- The `-2` option uses IEEE 802.3 ethernet transport

## III. Configuration Options
A possible reason deducted from the condition is that hardware timestamping is not configured properly, assuming hardware is not a problem, maybe the driver timestamping configuration is not properly set, due to older driver version.

To configure hardware timestamping on NIC, we may use `hwstamp_ctl` utility to set timestamping policy at driver level, even filter for spesific incoming packets to be timestamped (PTP messages). [Reference](https://ez.analog.com/dsp/software-and-development-tools/linux-for-adsp-sc5xx-processors/f/q-a/89484/how-to-enable-hw-timestamping-in-ptp)

| hwstamp_ctl options | AEEON NIC Capabilities |
| -------- | -------- |
|  |  |

This [linuxptp / Thread: \[Linuxptp-users\] HWSTAMP RX filter type](https://sourceforge.net/p/linuxptp/mailman/linuxptp-users/thread/8661e41d1f2847999c0f8e863b9e33df%40exch01.asrmicro.com/#msg37277793) explains why we need ptpv2-event and some reason ptpv2 may not work if absent.
**The requirements of successful ptp timestamp**
==NIC needs to support the feature + Kernel must have access to the registers + Driver needs to have access to port role/status==

:::success
In order to send and recieve timestamps correctly, we need to **run** `hwstamp_ctl` **before** running `ptp4l` to configure the driver before beginning TX/RX.

**The configuration solution:**
### For O-DU (master)

[osc@o-du linuxptp]$ hwstamp_ctl -i <interface> -r 9 -t 1
[osc@o-du linuxptp]$ sudo ./ptp4l -f ./configs/default.cfg -2 -i <interface> -m

`hwstamp_ctl` functions that we set :
- `-i` set for the specified interface
- `-r` set for NIC to timestamp the recieved **Delay_req** packet from slave (mode=14) *this is to get the t4 time*
- **Better to set to 9 to filter all incoming packet in ethernet**
- `-t` Enable hardware timestamping for outgoing packets (en=1)

### For O-RU (slave)

[osc@o-du linuxptp]$ hwstamp_ctl -i <interface> -r 9 -t 1
[osc@o-du linuxptp]$ sudo ./ptp4l -f ./configs/default_slave.cfg -2 -i <interface> -m

`hwstamp_ctl` functions that we set :
- `-i` set for the specified interface
- `-r` set for NIC to timestamp the recieved **Sync** packet (mode=13) *this is to get the t2 time*
- **Better to set to 9 to filter all incoming packetin ethernet**
- `-t` Enable hardware timestamping for outgoing packets (en=1)

\
**If Fails, consider updating the NIC driver version as the one recommended by OSC**
:::

:::warning
**Disclaimer**: Since the feature is not available for Software timestamping, I cannot experiment in VM and have not verify if the suggestion is sure to work. This note is based on study of mentioned works in reference.
:::

## IV. Troubleshooting Results
### A. Configure `hwstamp_ctl`
#### *Updated 04-08-2021*
Shuhua tried to apply the `hwstamp_ctl` configuration for O-DU and O-RU but no changes were made.

:::info
**Results:**

#### Intel server (RU as slave clock)

[oran@ee705-7-ip119 linuxptp]$ hwstamp_ctl -i enp24s0f1 -r 14 -t 1 
current settings: 
tx_type 1 
rx_filter 12 
SIOCSHWTSTAMP failed: Operation not permitted

rx-filter type 12 is for any layer, and any kind of packet
#### AAEON server (DU as master clock)

[oai@ee705-7-ip120 linuxptp]$ hwstamp_ctl -i enp134s0f0 -r 13 -t 1
current settings: 
tx_type 0 
rx_filter 0 
SIOCSHWTSTAMP failed: Operation not permitted

`SIOCSHWTSTAMP failed: Operation not permitted` needs sudo to access. But even if we try the command again on the O-RU that has the issue to follow O-DU configuration (timestamp all incoming packet) the results:

[oai@ee705-7-ip120 0-DU] sudo hwstamp_ctl -i enp134sof0 -r 12 -t 1
current settings:
tx type 0
rx filter 0
SIOCSHWTSTAMP failed: Numerical result out of range
The requested time stamping mode is not supported by the hardware.

The settings should not be 0, if both tx and rx = 0 then the NIC will not timestamp any packet and will not filter out timestamped packets, hence, offset cannot be calculated and both machines cannot be synced with PTP.
\
It turns out my hypothesis was wrong, the NIC in the O-RU have the same version as in O-DU but in O-DU it's working fine. So next we compare the two device's NIC info

| Intel server (RU as slave clock)| AAEON server (DU as master clock) |
| -------- | -------- |
| |  |

both are using the same hardware: **Intel Corporation Ethernet controller X710 for 10GbE SFP+** but they both have different firmware version (O-RU with the older version)
\
Kernel version should not be an issue because we are already using the OSC recommmended CentOS kernel version 3.10 (minimum for PTP is 3.00)
:::spoiler See details by OSC
> reference: [link](https://docs.o-ran-sc.org/projects/o-ran-sc-o-du-phy/en/latest/PTP-configuration_fh.html?highlight=kernel#ptp-for-linux-requirements)

:::

**In conclusion to the results** the only difference between the two is the **firmware version**. Since updating firmware may have bigger risk than updating driver, it is better to await review from Prof. Ray.

#### *Update 08/05/2021*
According to AAEON expert, Sam, the firmware is not the issue since after updating the firmware on AEEON site the tx_type and rx_filter stays 0. Awaiting response from Intel about this issue.
There *might* be a difference in the XL710 model used in the O-RU and O-DU but there is no way to verify this.

### B. Test Using Software Timestamping
#### *Update 12/08/2021*
After consulting with AEEON, the firmware update managed to enable the server to be set using `hwstamp_ctl` and so we tried setting the master to filter and create timestamps during PTP. The rx filter mode and tx timestamp mode is automatically changed after the update

:::spoiler See results
:::info
##### AEEON server O-DU (master)

##### Intel server O-RU (slave)

**Results:** Still not recieving timestamps
:::

:::success
#### Alternative solution : Software timestamping
Seeing the people in [linuxptp / Mailing Lists (sourceforge.net)](https://sourceforge.net/p/linuxptp/mailman/search/?q=recieved+pdelay_req+without+timestamp&mail_list=linuxptp-users&limit=100) which has the similar issue is related to hardware / kernel, we tried a command that works inside a VM simulation using **Software timestamping**
- timestamping will occur at OS / application layer
- get the time from system clock
- has relatively large error, but works for simulation

---

**Configuration**
##### AAEON server O-DU (master)

##### Intel server O-RU (slave)

Here the `-S` option is used to enable software timestamping and `-s` is to force slave only

:::

#### *Update 13/08/2021*
Found some hardware related factors (driver and machine kernel version) that may affect the PTPv2.

#### *Update 16/08/2021*
Attempting to update the kernel and NIC driver version to meet the requirements of PTP for OSC

**OSC setup configuration**

kernel version : vmlinuz-3.10.0-1062.12.1.rt56.1042.el7.x86_64

**NIC driver version**

driver: i40e
version: 2.10.19.82
firmware-version: 7.20 0x80007949 1.2585.0