---
tags: linux-rt, preempt-rt, real-time
---

# [Cyclictest](https://wiki.linuxfoundation.org/realtime/documentation/howto/tools/cyclictest/start) 實驗數據

:::success
Linux 核心版本和組態的標註方式:
1. `kernel 5.10.x-vanilla` 指 Linux 核心原始程式碼 v5.10.x，沒有程式碼變更，採預設核心組態
2. `kernel 5.10.x-rpi` 指 Raspberry Pi 基金會維護的 [rpi-5.10.y](https://github.com/raspberrypi/linux/tree/rpi-5.10.y) 分支，採預設核心組態
3. `kernel 5.10.78-rpi-rt56` 指在 [rpi-5.10.y](https://github.com/raspberrypi/linux/tree/rpi-5.10.y) 分支的基礎之上，附加 [preempt-rt](https://mirrors.edge.kernel.org/pub/linux/kernel/projects/rt/) 的變更，其中 `rt56` 是修訂版本號碼
4. 關於 preemption model 的標註，是在上述核心版本後面，以 `+` 附加，例如 `kernel 5.10.x-rpi+PREEMPT` 是 Raspberry Pi 基金會維護的 [rpi-5.10.y](https://github.com/raspberrypi/linux/tree/rpi-5.10.y) 分支搭配 `CONFIG_PREEMPT` 組態。有效的 preemption model 如下:
* `+NONE` 指 no kernel preemption
* `+PREEMPT` 指 `CONFIG_PREEMPT`
* `+PREEMPT_RT` 指 `CONFIG_PREEMPT_RT`
5. 關於 scheduling-clock ticks 的標註，也用 `+` 附加在上述核心版本和組態，有效的標示為:
* `+HZ` 指 `CONFIG_HZ_PERIODIC`
* `+NOHZ` 指 `CONFIG_NO_HZ`
* `+NOHZ_FULL` 指 `CONFIG_NO_HZ_FULL`
:::

## rt-test v1.0

### kernel 5.10.78+PREEMPT+NOHZ_FULL

$ taskset --cpu 0 sudo cyclictest -a 1 -i 500 -l 100000 -h 200 --histfile=output
# /dev/cpu_dma_latency set to 0us
policy: other/other: loadavg: 0.22 0.42 0.19 1/314 1269          

T: 0 ( 1269) P: 0 I:500 C: 100000 Min:      7 Act:   18 Avg:   19 Max:     113


### kernel 5.10.78-rt55+PREEMPT_RT+NOHZ_FULL

$ taskset --cpu 0 sudo cyclictest -a 1 -i 500 -l 100000 -h 200 --histfile=output
# /dev/cpu_dma_latency set to 0us
policy: other/other: loadavg: 0.98 0.85 0.39 1/340 1459          

T: 0 ( 1459) P: 0 I:500 C: 100000 Min:     12 Act:   27 Avg:   27 Max:    1020


---

## [rt-test](https://git.kernel.org/pub/scm/utils/rt-tests/rt-tests.git) v2.3 (latest)

### test script

#!/bin/bash

sudo cpufreq-set -g performance
sudo tuned-adm profile latency-performance

stress-ng -P 200 &
STRESS=$!

OUTPUT=output-`uname -r`-`date +%s`
sudo cyclictest -l 100000 -mS -h 400 --histfile=${OUTPUT}

kill -SIGTERM ${STRESS}

### kernel 5.10.78+PREEMPT_NONE+NOHZ_FULL


- Min Latencies ($\mu s$): 8, 8, 8, 8
- Avg Latencies ($\mu s$): 272, 196, 156, 147
- Max Latencies ($\mu s$): 52056, 36992, 17501, 15871

### kernel 5.10.78-rt56+PREEMPT_NONE+NOHZ_FULL


- Min Latencies ($\mu s$): 6, 8, 7, 7
- Avg Latencies ($\mu s$): 160, 154, 211, 254
- Max Latencies ($\mu s$): 18159, 20121, 27333, 35486

### kernel 5.10.78-rt56+PREEMPT_RT+NOHZ_FULL


- Min Latencies ($\mu s$): 9, 10, 10, 10
- Avg Latencies ($\mu s$): 73, 63, 62, 66
- Max Latencies ($\mu s$): 13320, 3568, 4892, 12673

### kernel 5.15.10-rt24-rpi+PREEMPT_RT+NOHZ_FULL


- Min Latencies ($\mu s$): 12, 12, 12, 11
- Avg Latencies ($\mu s$): 76, 69, 67, 67
- Max Latencies ($\mu s$): 1665, 3152, 3480, 3885

### 設定 cyclictest priority

#### test script

#!/bin/bash

sudo cpufreq-set -g performance
sudo tuned-adm profile latency-performance

stress-ng -P 200 &
STRESS=$!

OUTPUT=output-`uname -r`-`date +%s`
- sudo cyclictest -l 100000 -mS -h 400 --histfile=${OUTPUT}
+ sudo cyclictest -l 100000 -mS -p 99 -h 400 --histfile=${OUTPUT}

kill -SIGTERM ${STRESS}


- Min Latencies ($\mu s$): 8, 8, 8, 7
- Avg Latencies ($\mu s$): 11, 11, 12, 11
- Max Latencies ($\mu s$): 31, 28, 30, 30

---

## rt-tests v2.3 with [task-isolation](https://hackmd.io/@sysprog/Bkt9oOeKF) + [userspace-patches](https://people.redhat.com/~mtosatti/task-isol-v6-userspace-patches/)

核心開機時載入參數: `isolcpus=domain,nohz,2 nohz_full=2 irqaffinity=0,1,3`

### test script

#### For task-isolation disabled kernel

sudo tuned-adm profile latency-performance

stress-ng --taskset 2 -P 100 &
POLLING=$!

stress-ng --taskset 2 -T 50 &
TIMER=$!

OUTPUT=output-`uname -r`-`date +%s`
taskset 0xf sudo cyclictest --mainaffinity=0,1,3 -a 2 -l 100000 -i 200 -p 99 -m -h 400 --histfile=${OUTPUT}

kill -SIGTERM ${POLLING}
kill -SIGTERM ${TIMER}

#### For task-isolation enabled kernel

# chisol is from this patch: https://people.redhat.com/~mtosatti/task-isol-v6-userspace-patches/util-linux-chisol.patch

sudo tuned-adm profile latency-performance

stress-ng --taskset 2 -P 100 &
POLLING=$!

stress-ng --taskset 2 -T 50 &
TIMER=$!

OUTPUT=output-`uname -r`-`date +%s`
chisol -q vmstat_sync -I conf taskset 0xf sudo cyclictest --mainaffinity=0,1,3 -a 2 -l 100000 -i 200 -p 99 -m -h 400 --histfile=${OUTPUT}

kill -SIGTERM ${POLLING}
kill -SIGTERM ${TIMER}

### kernel 5.15.10-rpi+PREEMPT+NOHZ_FULL


- Min Latency ($\mu s$): 21
- Avg Latency ($\mu s$): 33
- Max Latency ($\mu s$): 85

### kernel 5.15.10-rpi+PREEMPT_RT+NOHZ_FULL


- Min Latency ($\mu s$): 6
- Avg Latency ($\mu s$): 14
- Max Latency ($\mu s$): 37

### kernel 5.15.10-rpi+PREEMPT+NOHZ_FULL


- Min Latency ($\mu s$): 21
- Avg Latency ($\mu s$): 33
- Max Latency ($\mu s$): 64

### kernel 5.15.10-rpi+PREEMPT_RT+NOHZ_FULL


- Min Latency ($\mu s$): 16
- Avg Latency ($\mu s$): 33
- Max Latency ($\mu s$): 73