kernel bisect - HackMD

kernel bisect === # 2024-09-12 Using [GitBisect]( https://wiki.debian.org/DebianKernel/GitBisect) instructions. Git bisect. Set bad to v6.6.15 and good to v6.7.12. Entered the bisection. It reported "a merge base must be tested". It did not report how many steps it would take. Apparently started with `[ffc253263a1375a65fa6c9f62a893e9767fbebfa] Linux 6.6`. The build failed with errors about signing and unabled to find some SSL bss...c file as reported by main.c. Internet says it was a known build issue. ``` ... HDRINST usr/include/asm/sockios.h HDRINST usr/include/asm/unistd_x32.h HDRINST usr/include/asm/bpf_perf_event.h HDRINST usr/include/asm/ioctl.h INSTALL debian/linux-libc-dev/usr/include SIGN debian/linux-image/lib/modules/6.6.15/kernel/arch/x86/events/amd/power.ko SIGN debian/linux-image/lib/modules/6.6.15/kernel/arch/x86/events/intel/intel-uncore.ko SIGN debian/linux-image/lib/modules/6.6.15/kernel/arch/x86/events/intel/intel-cstate.ko SIGN debian/linux-image/lib/modules/6.6.15/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko At main.c:298: - SSL error:FFFFFFFF80000002:system library::No such file or directory: ../crypto/bio/bss_file.c:67 - SSL error:10000080:BIO routines::no such file: ../crypto/bio/bss_file.c:75 sign-file: debian/linux-image/lib/modules/6.6.15/kernel/arch/x86/events/amd/power.ko SIGN debian/linux-image/lib/modules/6.6.15/kernel/arch/x86/events/rapl.ko SIGN debian/linux-image/lib/modules/6.6.15/kernel/arch/x86/kernel/msr.ko ``` If building kernels for debian, do not sign the modules. Set CONFIG_MODULE_SIG=n. # 2024-09-13 Realized that I should first reproduce Christian's findings. Building 6.6.15, I started the build, but it entered makeconfig-ish. Picked defauls for everything, except... AMD IOMMU Version 2 driver AMD_IOMMU_V2. Default was N, I picked m. ## Base Case I confirmed the base case test results from here https://salsa.debian.org/rocm-team/community/team-project/-/wikis/Navi-3x-QEMU-pass-through I'm using 20240709-1. (20240709-2 is out.) | version | test results | | --- | --- | | v6.6.15 | works | | v6.7.12 | broken | ## Bisection Will be about 10,000 commits to sift through. Estimated 14 steps to find the one commit where it breaks. | step | version | result | notes | | --- | --- | --- | --- | | 0 | v6.6.15 | good | | 1 | v6.6.0-14 | good | | 3 | v6.6.0-05843-g89ed67ef126c-16 | good | | 5 | v6.6.0-rc2-01403-g53dd920c1f47-18 | skip | boot failure (decompression failed) | | 6 | v6.5.0-rc7-01777-ge1133ac81176-20 | skip | boot failure (decompression failed) | | 7 | 6.5.0-rc7-01369-g80578f1641aa-20 | skip | boot failure (decompression failed) | | 8 | 6.6.0-rc2-00490-gc6a4f256aee1-21 | | boot failure. See boot fail notes, after this tested g53dd and it failed. | | 4 | v6.6.0-08649-g7d461b291e65-17 | bad | see 17 below | 2 | v6.6.0-111004-gedd8e84ae951-15 | bad | | 0 | v6.7.12 | bad | ## Boot failure Could be decompression failre. There's also "Failed to mount boot-efi.mount" in the dmesg. 6.6.15 books ok, but v6.6.0-01403-g53dd920c1f47 previously marked good is also now failing for similar reason. ## 17 - wacky result Was very slow to run. The VM has 8GB ram. I did not reboot the host before this test. ``` root@unstable-vm:~# rocminfo ROCk module is loaded [ 21.942246] amdgpu 0000:01:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x0000003B SMN_C2PMSG_82:0x00000000 [ 21.943553] amdgpu 0000:01:00.0: amdgpu: [SetDfCstate] failed! [ 21.944176] amdgpu 0000:01:00.0: amdgpu: Failed to disallow df cstate [ 25.444184] amdgpu 0000:01:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x0000003B SMN_C2PMSG_82:0x00000000 [ 25.445418] amdgpu 0000:01:00.0: amdgpu: Failed to retrieve enabled ppfeatures! [ 31.118066] amdgpu 0000:01:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x0000003B SMN_C2PMSG_82:0x00000000 [ 31.119320] amdgpu 0000:01:00.0: amdgpu: Failed to retrieve enabled ppfeatures! [ 31.146070] amdgpu 0000:01:00.0: Refused to change power state from D0 to D3hot [ 34.591795] amdgpu 0000:01:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x0000003B SMN_C2PMSG_82:0x00000000 [ 34.593076] amdgpu 0000:01:00.0: amdgpu: Failed to retrieve enabled ppfeatures! [ 34.596823] [drm] PCIE GART of 512M enabled (table at 0x0000008000000000). [ 34.597677] [drm] PSP is resuming... [ 34.621844] [drm] reserve 0x1300000 from 0x83fc000000 for PSP TMR [ 37.043429] [drm] psp gfx command SETUP_TMR(0x5) failed and response status is (0x0) [ 37.044249] [drm:psp_hw_start [amdgpu]] *ERROR* PSP load tmr failed! [ 37.045067] [drm:psp_resume [amdgpu]] *ERROR* PSP resume failed [ 37.045807] [drm:amdgpu_device_fw_loading [amdgpu]] *ERROR* resume of IP block <psp> failed -22 [ 37.046811] amdgpu 0000:01:00.0: amdgpu: amdgpu_device_ip_resume failed (-22). hsa api call failure at: ./rocminfo.cc:1204 Call returned HSA_STATUS_ERROR_OUT_OF_RESOURCES: The runtime failed to allocate the necessary resources. This error may also occur when the core runtime library needs to spawn threads or create internal OS-specific events. ``` ``` root@unstable-vm:~# free total used free shared buff/cache available Mem: 8131080 369888 7822336 580 157020 7761192 Swap: 0 0 0 ``` After rebooting the host, I get this error. ``` root@unstable-vm:~# rocminfo ROCk module is loaded error: kvm run failed Bad address RAX=0000000000002370 RBX=000000000000046f RCX=00030001014e5073 RDX=000000000000046e RSI=ffa0000030002370 RDI=ff1100011a580000 RBP=ff1100011ca6a350 RSP=ffa0000001803998 R8 =0003000000000073 R9 =ffa0000030000000 R10=ff11000165172bb8 R11=00000000000000f0 R12=ff1100011a580000 R13=0003000000000073 R14=ffa0000030000000 R15=000000000000046f RIP=ffffffffc0cea913 RFL=00000282 [--S----] CPL=0 II=0 A20=1 SMM=0 HLT=0 ES =0000 0000000000000000 00000000 00000000 CS =0010 0000000000000000 ffffffff 00a09b00 DPL=0 CS64 [-RA] SS =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS [-WA] DS =0000 0000000000000000 00000000 00000000 FS =0000 00007f44b42cc780 00000000 00000000 GS =0000 ff11000277b00000 00000000 00000000 LDT=0000 0000000000000000 00000000 00000000 TR =0040 fffffe0000079000 00004087 00008b00 DPL=0 TSS64-busy GDT= fffffe0000077000 0000007f IDT= fffffe0000000000 00000fff CR0=80050033 CR2=0000564a2e490598 CR3=0000000100a18005 CR4=00771ef0 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 DR6=00000000ffff0ff0 DR7=0000000000000400 EFER=0000000000000d01 Code=ff ff 00 00 48 21 c1 8d 04 d5 00 00 00 00 4c 09 c1 48 01 c6 <48> 89 0e 31 c0 e9 ee 97 a1 e9 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 ``` # Faster Builds * CT4000T700SSD5 - Crucial T700 4TB TLC NAND Flash PCIe Gen 5 x4 NVMe M.2 Internal SSD w/ Heatsink * CT4000T700SSD3 - Crucial T700 4TB TLC NAND Flash PCIe Gen 5 x4 NVMe M.2 Internal SSD https://linux-kernel.vger.kernel.narkive.com/4piRWaHb/tricks-to-speed-up-kernel-builds # 2024-09-16 The difference here is, I copy config from /boot/6.6.15 before every build. And I patch with this patch https://lore.kernel.org/all/3d34a965-ab9c-d549-0c63-c717ab5d2edc@tweek.dk/ based on this bug report https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1050582 ## Bisection | step | result | version | | ---:| ---:| --- | | 0 | good | v6.6.15 | | 1 | good | v6.6.0-14 | | 3 | good | v6.6.0-05843-g89ed67ef126c-16 | | 6 | good | 6.6.0-rc2-00675-g614351f41e8c-dirty | | 7 | good | 6.5.0-rc7-01718-g5710c6c0bda3-dirty | | 10 | good | 6.5.0-rc7-01763-gd849434977a8-dirty | | 13 | good | 6.5.0-rc7-01768-gcd956e753128-dirty | | 16 | good | 6.5.0-rc7-01769-g29495d81457a-dirty | | 15 | bad | 6.5.0-rc7-01770-gde59b69932e6-dirty | | 14 | bad | 6.5.0-rc7-01771-g41801c6b3042-dirty | | 12 | bad | 6.5.0-rc7-01774-gfa1f1cc09d58-dirty | | 11 | bad | 6.5.0-rc7-01786-g3a41a3982b1e-dirty | | 9 | bad | 6.5.0-rc7-01809-g0021d70a0654-dirty | | 8 | bad | 6.5.0-rc7-01900-gb3e73b5a8f8a-dirty | | 5 | bad | 6.6.0-rc2-01403-g53dd920c1f47-dirty | | 4 | bad | v6.6.0-08649-g7d461b291e65-17 | | 2 | bad | v6.6.0-111004-gedd8e84ae951-15 | | 0 | bad | v6.7.12 | # b3e7 rocminfo Did not reboot host before this test. ``` root@unstable-vm:~# rocminfo ROCk module is loaded [ 7889.130929] amdgpu 0000:01:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x0000003B SMN_C2PMSG_82:0x00000000 [ 7889.132293] amdgpu 0000:01:00.0: amdgpu: Failed to retrieve enabled ppfeatures! [ 7889.133434] [drm] PCIE GART of 512M enabled (table at 0x0000008000000000). [ 7889.134396] [drm] PSP is resuming... [ 7889.158531] [drm] reserve 0x1300000 from 0x83fc000000 for PSP TMR [ 7891.564656] [drm] psp gfx command SETUP_TMR(0x5) failed and response status is (0x0) [ 7891.565461] [drm:psp_hw_start [amdgpu]] *ERROR* PSP load tmr failed! [ 7891.566267] [drm:psp_resume [amdgpu]] *ERROR* PSP resume failed [ 7891.567001] [drm:amdgpu_device_fw_loading [amdgpu]] *ERROR* resume of IP block <psp> failed -22 [ 7891.568021] amdgpu 0000:01:00.0: amdgpu: amdgpu_device_ip_resume failed (-22). hsa api call failure at: ./rocminfo.cc:1204 Call returned HSA_STATUS_ERROR_OUT_OF_RESOURCES: The runtime failed to allocate the necessary resources. This error may also occur when the core runtime library needs to spawn threads or create internal OS-specific events. ``` on second host reboot ``` root@unstable-vm:~# rocminfo ROCk module is loaded error: kvm run failed Bad address RAX=0000000000002360 RBX=000000000000046d RCX=0003000115d83073 RDX=000000000000046c RSI=ff7b151130002360 RDI=ff418701d4e00000 RBP=ff4187019c3db030 RSP=ff7b151100497998 R8 =0003000000000073 R9 =ff7b151130000000 R10=ff418701dc8c01b8 R11=ff4187018085ac60 R12=ff418701d4e00000 R13=0003000000000073 R14=ff7b151130000000 R15=000000000000046d RIP=ffffffffc0eeab13 RFL=00000286 [--S--P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 ES =0000 0000000000000000 00000000 00000000 CS =0010 0000000000000000 ffffffff 00a09b00 DPL=0 CS64 [-RA] SS =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS [-WA] DS =0000 0000000000000000 00000000 00000000 FS =0000 00007f4d64ccd780 00000000 00000000 GS =0000 ff418702f7b00000 00000000 00000000 LDT=0000 0000000000000000 00000000 00000000 TR =0040 fffffe64511e8000 00004087 00008b00 DPL=0 TSS64-busy GDT= fffffe64511e6000 0000007f IDT= fffffe0000000000 00000fff CR0=80050033 CR2=0000563f11c84598 CR3=000000012a28a005 CR4=00771ee0 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 DR6=00000000ffff0ff0 DR7=0000000000000400 EFER=0000000000000d01 Code=ff ff 00 00 48 21 c1 8d 04 d5 00 00 00 00 4c 09 c1 48 01 c6 <48> 89 0e 31 c0 e9 a3 28 30 d1 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 ``` # fa1f ``` root@unstable-vm:/shared# mkdir output root@unstable-vm:/shared# cd [ 24.164282] amdgpu 0000:01:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x0000003B SMN_C2PMSG_82:0x00000000 [ 24.165499] amdgpu 0000:01:00.0: amdgpu: [SetDfCstate] failed! [ 24.166076] amdgpu 0000:01:00.0: amdgpu: Failed to disallow df cstate output root@unstable-vm:/shared/output# rocminfo[ 27.616141] amdgpu 0000:01:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x0000003B SMN_C2PMSG_82:0x00000000 [ 27.617366] amdgpu 0000:01:00.0: amdgpu: Failed to retrieve enabled ppfeatures! > rocminf[ 33.235786] amdgpu 0000:01:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x0000003B SMN_C2PMSG_82:0x00000000 [ 33.237142] amdgpu 0000:01:00.0: amdgpu: Failed to retrieve enabled ppfeatures! [ 33.263790] amdgpu 0000:01:00.0: Refused to change power state from D0 to D3hot root@unstable-vm:/shared/output# rocminfo ROCk module is loaded [ 40.490323] amdgpu 0000:01:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x0000003B SMN_C2PMSG_82:0x00000000 [ 40.491669] amdgpu 0000:01:00.0: amdgpu: Failed to retrieve enabled ppfeatures! [ 40.493801] [drm] PCIE GART of 512M enabled (table at 0x0000008000000000). [ 40.495668] [drm] PSP is resuming... [ 40.519782] [drm] reserve 0x1300000 from 0x83fc000000 for PSP TMR [ 42.947146] [drm] psp gfx command SETUP_TMR(0x5) failed and response status is (0x0) [ 42.947944] [drm:psp_hw_start [amdgpu]] *ERROR* PSP load tmr failed! [ 42.948745] [drm:psp_resume [amdgpu]] *ERROR* PSP resume failed [ 42.949440] [drm:amdgpu_device_fw_loading [amdgpu]] *ERROR* resume of IP block <psp> failed -22 [ 42.950455] amdgpu 0000:01:00.0: amdgpu: amdgpu_device_ip_resume failed (-22). hsa api call failure at: ./rocminfo.cc:1204 Call returned HSA_STATUS_ERROR_OUT_OF_RESOURCES: The runtime failed to allocate the necessary resources. This error may also occur when the core runtime library needs to spawn threads or create internal OS-specific events. ``` # Final Result ``` ❯ git bisect good de59b69932e64d77445d973a101d81d6e7e670c6 is the first bad commit commit de59b69932e64d77445d973a101d81d6e7e670c6 Author: Alex Deucher <alexander.deucher@amd.com> Date: Wed Sep 20 13:27:58 2023 -0400 drm/amdgpu/gmc: set a default disable value for AGP To disable AGP, the start needs to be set to a higher value than the end. Set a default disable value for the AGP aperture and allow the IP specific GMC code to enable it selectively be calling amdgpu_gmc_agp_location(). Reviewed-by: Christian König <christian.koenig@amd.com> Signed-off-by: Alex Deucher <alexander.deucher@amd.com> drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c | 27 ++++++++++++++++------- drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h | 2 ++ drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 3 +++ drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c | 3 ++- drivers/gpu/drm/amd/amdgpu/gmc_v11_0.c | 3 ++- drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c | 4 ++-- drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c | 4 ++-- drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c | 4 ++-- drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c | 3 ++- drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c | 2 +- 10 files changed, 37 insertions(+), 18 deletions(-) ```

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.