# [AIdrifter CS 浮生筆錄](https://hackd.io/s/rypeUnYSb) : kenrel CoreDump - 強者我同事的教學 ## 如何分析 kernel panic - 1. 拿到 kernel panic 的 log 如下,需要知道紅色的部分,也就是 __pc__ 和 __lr__ 的。__(line 12 and 13)__ - PC is at PMRegisterTovendor_driver+`0xf0/0x210` [vendor_driver_ap] - 以本例子來看是死在 `vendor_driver_ap.ko` 裡面 ```bash=1 # address [ 40.625639] Unable to handle kernel NULL pointer dereference at virtual address 00000000 [ 40.633793] pgd = ffffffc054b37000 [ 40.637329] [00000000] *pgd=00000000d6e5f003[ 40.641446] , *pud=00000000d6e5f003 , *pmd=00000000d6ca1003[ 40.646996] , *pte=0000000000000000 [ 40.650508] [ 40.652022] Internal error: Oops: 96000047 [#1] PREEMPT SMP [ 40.657600] Modules linked in: mali_kbase(O) kds(O) sun_fbdev_mi(O) mik(O) misck(PO) HEAAC_ENC(PO) mwgifker(PO) xcker(PO) iniparser(O) vendor_driver_ap(O) Drv_xc(O) ntfs_3g(O) firmware_class cfg80211 [ 40.674932] CPU: 1 PID: 1521 Comm: l8v_linux1 Tainted: P O 4.9.32+ #1 [ 40.682506] Hardware name: l8v (DT) [ 40.685998] task: ffffffc04e4c0000 task.stack: ffffffc059554000 [ 40.692679] PC is at PMRegisterTovendor_driver+0xf0/0x210 [vendor_driver_ap] # <---PC address [ 40.698924] LR is at PMRegisterTovendor_driver+0xe8/0x210 [vendor_driver_ap] # <---LR address [ 40.704503] pc : [<ffffff8000e197f0>] lr : [<ffffff8000e197e8>] pstate: 80000145 [ 40.711903] sp : ffffffc059557620 ``` - 2. 在 kernel console 下面下 `lsmod` 會出現各 ko 目前 insert 的狀況 - 以本例子可以看到 __vendor_driver_ap.ko__ 在 kernel 中被 insert 到 `0xffffff8000b35000` 這個 address。__(line 10)__ - 搭配 kernel log 所取得的 panic pc 減去 __vendor_driver_ap.ko__ 的 base address,可以得到 shift 的 address。 `0xffffff8000e197f0 - 0xffffff8000b35000 = 0x2E47F0` ```bash=1 [2/9115] mali_kbase 419731 0 - Live 0xffffff8001bbc000 (O) kds 7100 1 mali_kbase, Live 0xffffff800099e000 (O) sun_fbdev_mi 20172 1 - Live 0xffffff8000998000 (O) mik 3846296 100 sun_fbdev_mi, Live 0xffffff8001ea3000 (O) misck 726023 1 mik, Live 0xffffff8001dd3000 (PO) HEAAC_ENC 202166 1 mik, Live 0xffffff8001d97000 (PO) mwgifker 139739 1 mik, Live 0xffffff8001d67000 (PO) xcker 461156 1 mik, Live 0xffffff8001cc5000 (PO) iniparser 19631 1 mik, Live 0xffffff8001cbb000 (O) vendor_driver_ap 17322107 5 mik,misck,HEAAC_ENC,mwgifker,xcker, Live 0xffffff8000b35000 (O) # <-- vendor_driver_ap.ko here Drv_xc 1113463 1 vendor_driver_ap, Live 0xffffff80009ff000 (O) ntfs_3g 291040 0 - Live 0xffffff80009a6000 (O) firmware_class 6732 0 - Live 0xffffff80009a1000 cfg80211 222761 0 - Live 0xffffff8000960000 ``` - 3. 使用 add2line 反推死掉 .c 的位置(需用 .o) - 可以知道死在 __halPM.c__ 第**555**行的地方 ```bash aarch64-linux-gnu-addr2line -e build/vendor_driver_ap.o 0x2E47F0 ``` ```info /vendor_driver_src/build//../../halPM.c:555 ``` ## 如何製造 BT,看call trace - 可以在 kernel space 任何地方呼叫 `dump_stack()` 去 dump 該 function 被呼叫的 BT 如下,此 BT 不會造成任何當機的問題 - 但是會在 kernel log 打印 BT 的訊息。 ```bash=1 [ 32.327542] [<ffffff800808e98c>] dump_backtrace+0x0/0x1a4 [ 32.327545] [<ffffff800808eb44>] show_stack+0x14/0x1c [ 32.327550] [<ffffff80084209e8>] dump_stack+0x8c/0xb0 [ 32.328080] [<ffffff8000e197ec>] PMRegisterTovendor_driver+0xec/0x208 [vendor_driver_ap] [ 32.328431] [<ffffff8000e1a0ac>] HAL_PM_SetSRAMOffsetForMCU+0xc4/0x118 [vendor_driver_ap] [ 32.328757] [<ffffff8000e182e0>] Drv_PM_SetSRAMOffsetForMCU+0x2c/0x34 [vendor_driver_ap] [ 32.330112] [<ffffff8002078898>] _MI_PM_RunPmOnSram+0x6c0/0x7c8 [mik] [ 32.331425] [<ffffff8001fe14ec>] _MI_PM_Pm51Run+0x614/0xf80 [mik] [ 32.332673] [<ffffff8001fe2994>] MI_PM_SetStrStage+0x26c/0x4d8 [mik] [ 32.333895] [<ffffff8001ec3d88>] MI_DEV_PM_Suspend+0x138/0x220 [mik] [ 32.335112] [<ffffff8001ea3020>] _sun_drv_suspend+0xc/0x14 [mik] [ 32.335124] [<ffffff80085050cc>] platform_pm_suspend+0x4c/0x50 [ 32.335129] [<ffffff8008510150>] dpm_run_callback.isra.13+0x20/0x6c [ 32.335134] [<ffffff8008510bc0>] __device_suspend+0x138/0x274 [ 32.335138] [<ffffff8008511dc4>] dpm_suspend+0x114/0x240 [ 32.335143] [<ffffff80085121c8>] dpm_suspend_start+0x6c/0x78 [ 32.335151] [<ffffff8008108f88>] suspend_devices_and_enter+0xac/0x1bc [ 32.335156] [<ffffff8008109394>] pm_suspend+0x2fc/0x3d4 [ 32.335160] [<ffffff8008107d78>] state_store+0xa0/0x110 [ 32.335167] [<ffffff8008423884>] kobj_attr_store+0x14/0x24 [ 32.335173] [<ffffff8008282918>] sysfs_kf_write+0x40/0x50 [ 32.335177] [<ffffff8008281cf8>] kernfs_fop_write+0xb0/0x1d0 [ 32.335184] [<ffffff800820413c>] __vfs_write+0x28/0x110 [ 32.335188] [<ffffff8008204e74>] vfs_write+0xa0/0x170 [ 32.335191] [<ffffff8008205fa0>] SyS_write+0x44/0xa0 [ 32.335197] [<ffffff8008082eb0>] el0_svc_naked+0x24/0x28