# RCU Torture ###### tags: `rcu` ## RCU Torture Test #### paul's materials - [Stupid RCU Tricks: A tour through rcutorture](https://paulmck.livejournal.com/61432.html) RCU Torture guideline - [So You Want To Torture RCU?](https://indico.lpc.events/event/11/contributions/897/attachments/792/1703/rcutorture.2021.09.22a.pdf) https://youtu.be/8Mlh4g4pJnw paul's slide & speech #### Doc - [RCU Torture Test Operation (docs)](https://docs.kernel.org/RCU/torture.html) - [Documentation / RCU / torture.txt](https://android.googlesource.com/kernel/msm/+/f5335159eed416b26b7c8a5a4e8820f97dc1ad19/Documentation/RCU/torture.txt) --- ### Test environment - `sudo lshw -short -class processor` ``` H/W path Device Class Description ===================================================================== /0/31 processor AMD Ryzen 5 5600X 6-Core Processor ``` - `uname -a` ``` Linux ccs100203-System-Product-Name 5.15.0-41-generic #44~20.04.1-Ubuntu SMP Fri Jun 24 13:27:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux ``` ---- ### Running rcutorture in a Guest OS > [Stupid RCU Tricks: So you want to torture RCU?](https://paulmck.livejournal.com/57769.html) > [Format of rcutorture Summary](https://mirrors.edge.kernel.org/pub/linux/kernel/people/paulmck/Answers/RCU/rcutorture-summary.html) > [rcutorture scripting](https://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git/tree/tools/testing/selftests/rcutorture) - run up to 12 CPUs worth of rcutorture scenarios concurrently `tools/testing/selftests/rcutorture/bin/kvm.sh --cpus 12` ``` --- Fri 29 Jul 2022 06:34:57 AM CST Test summary: Results directory: /home/ccs100203/Desktop/TMPP/test_rcu/linux-5.15.41/tools/testing/selftests/rcutorture/res/2022.07.29-01.55.59 tools/testing/selftests/rcutorture/bin/kvm.sh --cpus 12 RUDE01 ------- 13143 GPs (7.30167/s) [tasks-rude: g0 f0x0 total-gps=0] SRCU-N ------- 230632 GPs (128.129/s) [srcu: g2983724 f0x0 total-gps=2983724] SRCU-P ------- 100984 GPs (56.1022/s) [srcud: g734336 f0x0 total-gps=734336] SRCU-T ------- 315973 GPs (175.541/s) [srcu: g63496 f0x0 total-gps=63496] SRCU-U ------- 874484 GPs (485.824/s) [srcud: g38358 f0x0 total-gps=38358] TASKS01 ------- 6149 GPs (3.41611/s) [tasks: g0 f0x0 total-gps=0] TASKS02 ------- 6780 GPs (3.76667/s) [tasks: g0 f0x0 total-gps=0] TASKS03 ------- 6066 GPs (3.37/s) [tasks: g0 f0x0 total-gps=0] TINY01 ------- 257562 GPs (143.09/s) [rcu: g0 f0x0 total-gps=0] n_max_cbs: 70301 TINY02 ------- 267704 GPs (148.724/s) [rcu: g0 f0x0 total-gps=0] n_max_cbs: 2810 TRACE01 ------- 7491 GPs (4.16167/s) [tasks-tracing: g0 f0x0 total-gps=0] TRACE02 ------- 83341 GPs (46.3006/s) [tasks-tracing: g0 f0x0 total-gps=0] TREE01 ------- 33812 GPs (18.7844/s) [rcu: g249101 f0x0 total-gps=62559] TREE02 ------- 82800 GPs (46/s) [rcu: g647629 f0x0 total-gps=162190] n_max_cbs: 227970 TREE03 ------- 42677 GPs (23.7094/s) [rcu: g831845 f0x0 total-gps=208243] n_max_cbs: 2099217 CPU count limited from 16 to 12 TREE04 ------- 71396 GPs (39.6644/s) [rcu: g559645 f0x0 total-gps=140201] n_max_cbs: 652148 TREE05 ------- 72440 GPs (40.2444/s) [rcu: g538305 f0x0 total-gps=134859] n_max_cbs: 154249 TREE07 ------- 59138 GPs (32.8544/s) [rcu: g448573 f0x0 total-gps=112429] n_max_cbs: 488673 CPU count limited from 16 to 12 TREE09 ------- 220674 GPs (122.597/s) [rcu: g3424197 f0x0 total-gps=856329] n_max_cbs: 2162264 --- Done at Fri 29 Jul 2022 06:34:58 AM CST (4:38:59) exitcode 0 ``` #### TREE09 - 220674 GPs: > This is the number of grace periods requested by the `rcu_torture_writer()` function, which drives updates. `rcu_torture_writer()` 請求 grace period 的次數 #### rcu: g3424197 > This number is a count of the actual number of grace periods that the underlying RCU flavor performed, which will normally be quite a bit larger than the number requested by the rcu_torture_writer() function. 底層的 RCU 實際請求 grace period 的次數 #### f0x0: > This is a flavor specific flag value, in this case indicating whether or not there is an outstanding request to the grace-period kthread. For example, had there been a request for an additional grace period, the low-order bit would have been set. a specific flag, e.g. set 最低位的 bit 代表額外請求一次 grace period #### n_max_cbs: 2162264 > This is the maximum number of RCU callbacks that rcutorture's callback-flooding testing was able queue up. 在 rcutorture's callback-flooding testing 中能夠排進隊列中的最大 RCU callbacks 數量 ---- ### Locating Test Failures No errors in my above test ``` $ ./kvm-find-errors.sh ../res/2022.07.29-01.55.59/ No build errors. No errors in console logs. ``` #### Did it mean RCU is perfect in the Linux kernel? Absolutely not > If rcutorture produces a failure-free run, that is a failure on the part of rcutorture. After all, there are bugs in there somewhere, and rcutorture failed to find them! - Just the test program cannot find other bugs ![](https://i.imgur.com/HbAQdVu.png =600x) #### Some approaches help us to find them (bugs) - Increase the duration of test program, like `--duration 720` (720 mins) - Enlist the help of other in-kernel torture features e.g. `--kconfig "CONFIG_DEBUG_LOCK_ALLOC=y CONFIG_PROVE_LOCKING=y"` - Adding the kernel address sanitizer for test > The aid of the kernel address sanitizer (KASAN) can be enlisted using the --kasan argument: `tools/testing/selftests/rcutorture/bin/kvm.sh --cpus 12 --kasan` - Still no errors with kernel address sanitizer ``` --- Mon 01 Aug 2022 01:29:41 PM CST Test summary: Results directory: /home/ccs100203/Desktop/TMPP/test_rcu/linux-5.15.41/tools/testing/selftests/rcutorture/res/2022.08.01-08.32.17 tools/testing/selftests/rcutorture/bin/kvm.sh --cpus 12 --kasan RUDE01 ------- 13119 GPs (7.28833/s) [tasks-rude: g0 f0x0 total-gps=0] SRCU-N ------- 230845 GPs (128.247/s) [srcu: g3008944 f0x0 total-gps=3008944] SRCU-P ------- 103449 GPs (57.4717/s) [srcud: g736724 f0x0 total-gps=736724] SRCU-T ------- 313716 GPs (174.287/s) [srcu: g43648 f0x0 total-gps=43648] SRCU-U ------- 869683 GPs (483.157/s) [srcud: g2460 f0x0 total-gps=2460] TASKS01 ------- 6225 GPs (3.45833/s) [tasks: g0 f0x0 total-gps=0] TASKS02 ------- 6773 GPs (3.76278/s) [tasks: g0 f0x0 total-gps=0] TASKS03 ------- 6191 GPs (3.43944/s) [tasks: g0 f0x0 total-gps=0] TINY01 ------- 265657 GPs (147.587/s) [rcu: g0 f0x0 total-gps=0] n_max_cbs: 2542 TINY02 ------- 283918 GPs (157.732/s) [rcu: g0 f0x0 total-gps=0] n_max_cbs: 403 TRACE01 ------- 7536 GPs (4.18667/s) [tasks-tracing: g0 f0x0 total-gps=0] TRACE02 ------- 84910 GPs (47.1722/s) [tasks-tracing: g0 f0x0 total-gps=0] TREE01 ------- 46760 GPs (25.9778/s) [rcu: g328857 f0x0 total-gps=82491] TREE02 ------- 103878 GPs (57.71/s) [rcu: g804297 f0x0 total-gps=201349] n_max_cbs: 13749 TREE03 ------- 55818 GPs (31.01/s) [rcu: g894053 f0x0 total-gps=223781] n_max_cbs: 53236 CPU count limited from 16 to 12 TREE04 ------- 90969 GPs (50.5383/s) [rcu: g690861 f0x0 total-gps=173004] n_max_cbs: 13270 TREE05 ------- 96166 GPs (53.4256/s) [rcu: g711205 f0x0 total-gps=178083] n_max_cbs: 26804 TREE07 ------- 81675 GPs (45.375/s) [rcu: g603829 f0x0 total-gps=151241] n_max_cbs: 14295 CPU count limited from 16 to 12 TREE09 ------- 226178 GPs (125.654/s) [rcu: g3473152 f0x0 total-gps=868535] n_max_cbs: 45198 --- Done at Mon 01 Aug 2022 01:29:42 PM CST (4:57:25) exitcode 0 ``` > [Concurrency bugs should fear the big bad data-race detector (part 1)](https://lwn.net/Articles/816850/) > [Concurrency bugs should fear the big bad data-race detector (part 2)](https://lwn.net/Articles/816854/) ---- ### Race condition testing > [Stupid RCU Tricks: Failure Probability and CPU Count](https://paulmck.livejournal.com/58187.html) > One problem is that, rcutorture being what it is, that bug is likely a race condition and it likely takes longer than you would like to reproduce. Assuming that it reproduces at all. > we are trying to make a race condition happen more frequently, it is only natural to try tweaking the number of CPUs. 有時候 bug 是因為 race condition 而發生的，所以在測試時想要盡可能的重現他 那麼要如何讓 race condition 發生的更頻繁呢？ 最直觀的方法就是調整測試時使用的 CPU 數量，在一個 rctorture scenario 盡可能使用所有的 CPU。 但即使在一個 scenario 下設定 `CONFIG_NR_CPUS=64` 也不能保證所有 CPU 執行的機率是相等的，因為在某些情況下，像是 race condition 發生在 RCU's grace-period kthread 之間，那同時就只會執行在最多一個 workqueue kthread 上。 在上述的情況下，可以針對特定 scenario (rcutorture/configs/rcu/TREE03) 設定成 `CONFIG_NR_CPUS=2`，並在 `kvm.sh` 中指定 `--allcpus`，讓一個 scenario 中最多就是兩個 CPU 參與。 > For a race condition that rcutorture uncovered during the v5.8 merge window, running one large rcutorture instance instead of 14 smaller ones (very) roughly doubled the probability of locating the race condition. 一般來說，執行一個大的 rcutorture instance 會比執行 14 個小的 instances 多一倍的機率去定位出 race condition TODO, [Stupid RCU Tricks: Making Race Conditions More Probable](https://paulmck.livejournal.com/62071.html) --- ## Test Analyzing ### Configs > [Hierarchical RCU](https://lwn.net/Articles/305782/#Testing) > [Kernel configuration parameters for RCU](https://lwn.net/Articles/777214/) > [rcutorture/doc/](https://github.com/torvalds/linux/tree/master/tools/testing/selftests/rcutorture/doc) - CONFIG_CLASSIC_RCU: Classic RCU. - CONFIG_PREEMPT_RCU: Preemptable (real-time) RCU. - CONFIG_TREE_RCU: Classic RCU for huge SMP systems. - CONFIG_RCU_FANOUT: Number of children for each rcu_node. - CONFIG_RCU_FANOUT_EXACT: Balance the rcu_node tree. - CONFIG_HOTPLUG_CPU: Allow CPUs to be offlined and onlined. - CONFIG_NO_HZ: Enable dyntick-idle mode. - CONFIG_SMP: Enable multi-CPU operation. - CONFIG_RCU_CPU_STALL_DETECTOR: Enable RCU to detect when CPUs go on -extended quiescent-state vacations. - CONFIG_RCU_TRACE: Generate RCU trace files in debugfs. - CONFIG_RCU_EXPERT: 有些 kconfig 需要 set 此選項才能開啟 - CONFIG_RCU_NOCB_CPU=y: fortuitously improves RCU's energy efficiency by eliminating wakeups due to RCU callback processing. ### TORTURE MODULE PARAMETERS > [Documentation / RCU / torture.txt](https://android.googlesource.com/kernel/msm/+/f5335159eed416b26b7c8a5a4e8820f97dc1ad19/Documentation/RCU/torture.txt) ---- ### Torture testcases > [rcutorture/configs/rcu/](https://github.com/torvalds/linux/tree/master/tools/testing/selftests/rcutorture/configs/rcu) #### TREE01 CONFIG_PREEMPT=y 代表 CONFIG_PREEMPT_RCU，會用 tree-based RCU implementation 來做到 real-time and low-latency SMP 的建構，可以容納大量的 CPU 除了 preempt 的部份，此測試也涵蓋到了 cpu offline/online 的測試 ``` CONFIG_SMP=y CONFIG_PREEMPT=y CONFIG_HOTPLUG_CPU=y CONFIG_RCU_NOCB_CPU=y ``` #### TREE02 同 TREE01 會有 preempt，但測試的點是在於改變了 `rcu_node` 的 FANOUT 數量，測試調整 data-structure layout 的結果 ``` CONFIG_SMP=y CONFIG_PREEMPT=y CONFIG_NR_CPUS=8 CONFIG_RCU_FANOUT=3 CONFIG_RCU_FANOUT_LEAF=3 CONFIG_RCU_NOCB_CPU=n ``` #### TREE03 > CONFIG_RCU_BOOST=y enables RCU priority boosting. This could be considered a debugging option, but it is one that pertains primarily to real-time kernels, so is included in the real-time section. This Kconfig parameter causes blocked RCU readers to be priority-boosted in order to avoid indefinite prolongment of the current RCU grace period. 同 TREE02 會有 preempt，測試的點除了改變 FANOUT 之外，也增加了 CONFIG_RCU_BOOST=y，對於 real-time kernel 中被 block 的 RCU reader，可以提高他的 priority ``` CONFIG_NR_CPUS=16 CONFIG_HOTPLUG_CPU=y CONFIG_RCU_FANOUT=2 CONFIG_RCU_FANOUT_LEAF=2 CONFIG_RCU_NOCB_CPU=n CONFIG_RCU_BOOST=y ``` #### TREE10 > [rcutorture: Add 100-CPU configuration](https://github.com/torvalds/linux/commit/e0714247373b9c0253b002573f63f3e9698b7b30) > CONFIG_PREEMPT=n and CONFIG_SMP=y implies CONFIG_TREE_RCU, thus selecting the non-preemptible tree-based RCU implementation that is appropriate for server-class SMP builds. 大部分的測試都是在 small-system torture 的情況下，這個是測試多個 CPU 下的 RCU CONFIG_PREEMPT=n and CONFIG_SMP=y 代表選擇了 CONFIG_TREE_RCU，這很適合用在 server-class SMP 的建構，可以容納大量的 CPU ``` CONFIG_SMP=y CONFIG_NR_CPUS=56 CONFIG_PREEMPT_NONE=y CONFIG_PREEMPT_VOLUNTARY=n CONFIG_PREEMPT=n ``` #### TINY01 CONFIG_PREEMPT=n and CONFIG_SMP=n 代表選擇了 CONFIG_TINY_RCU，是屬於 non-preemptible uniprocessor (UP) RCU implementation，適合用在 non-real-time UP 的建構，所佔用的 memory 是 kernel 中最小的 ``` CONFIG_SMP=n CONFIG_PREEMPT=n ``` #### TASKS01 CONFIG_TASKS_RCU=y 測試 TASK flavor 的 RCU ``` CONFIG_SMP=y CONFIG_PREEMPT=y CONFIG_NR_CPUS=4 CONFIG_HOTPLUG_CPU=y CONFIG_TASKS_RCU=y ``` #### TASKS02 在 CONFIG_PREEMPT=n 的情況下，TASK flavor implementation 所用到的 call_rcu_tasks() 要對應到 call_rcu()，而 synchronize_rcu_tasks() 要對應到 synchronize_rcu() ``` CONFIG_SMP=n CONFIG_PREEMPT=n #CHECK#CONFIG_TASKS_RCU=y ``` #### SRCU-N 測試在 SMP 開啟，而不能 preempt 下的 TREE SRCU 是否正常運作 ``` CONFIG_SMP=y CONFIG_NR_CPUS=4 CONFIG_HOTPLUG_CPU=y CONFIG_PREEMPT_NONE=y CONFIG_PREEMPT_VOLUNTARY=n CONFIG_PREEMPT=n ``` #### SRCU-P 測試 TREE SRCU 在 three-level srcu_node tree (in tall tree) 並且可以被 preempt 的情況下是否正常 ``` CONFIG_SMP=y CONFIG_NR_CPUS=8 CONFIG_RCU_EXPERT=y CONFIG_RCU_FANOUT=2 CONFIG_RCU_FANOUT_LEAF=2 CONFIG_PREEMPT=y ``` #### SRCU-U 測試 TINY SRCU 是否能正常運作 ``` CONFIG_SMP=n CONFIG_PREEMPT_NONE=y CONFIG_PREEMPT_VOLUNTARY=n CONFIG_PREEMPT=n ``` --- ## Reference - [Stupid RCU Tricks: A tour through rcutorture](https://paulmck.livejournal.com/61432.html) paul's introduction articles about rcu torture - [RCU torture testing](https://lwn.net/Articles/154107/) lwn article of adding `CONFIG_RCU_TORTURE_TEST` into Linux kernel by paul - [RCU Torture Test Operation](https://www.kernel.org/doc/Documentation/RCU/torture.txt) **Document** of RCU Torture Test - [Documentation / RCU / torture.txt](https://android.googlesource.com/kernel/msm/+/f5335159eed416b26b7c8a5a4e8820f97dc1ad19/Documentation/RCU/torture.txt) the same doc as the above link - [RCU Torture Test Operation (docs)](https://docs.kernel.org/RCU/torture.html) another document of RCU Torture Test - [/kernel/rcu/rcutorture.c](https://elixir.bootlin.com/linux/latest/source/kernel/rcu/rcutorture.c) - [linux-test-project/ltp - rcu_torture.sh](https://github.com/linux-test-project/ltp/blob/master/testcases/kernel/device-drivers/rcu/rcu_torture.sh) - [Verification Challenges](https://paulmck.livejournal.com/65721.html) You would like to do some formal verification of C code? Or you would like a challenge for your formal-verification tool? Either way, here you go!