Raspberry pi KGDB using FT2232H === KGDB顧名思義就是Kernel GDB，可以除錯Linux kernel。想要試著實做的原因是之前在COSCUP 2021聽了如何使用KGDB trace linux kernel的conference。一直想來實做看看，這篇紀錄我的實作過程。 [參考 COSCUP 2021 KGDB conference](https://hackmd.io/@coscup/Bk8qLaw0d/%2F%40coscup%2FrJu8I6PCO) 本文章使用FT2232H連接Raspberry pi的JTAG，並藉由OpenOCD和KGDB trace Linux kernel開機過程。以function kernel_init()為本次trace的範例。[參考](https://elixir.bootlin.com/linux/latest/source/init/main.c#L1480) ```Clike= static int __ref kernel_init(void *unused) { int ret; /* * Wait until kthreadd is all set-up. */ wait_for_completion(&kthreadd_done); kernel_init_freeable(); /* need to finish all async __init code before freeing the memory */ async_synchronize_full(); kprobe_free_init_mem(); ftrace_free_init_mem(); kgdb_free_init_mem(); free_initmem(); mark_readonly(); /* * Kernel mappings are now finalized - update the userspace page-table * to finalize PTI. */ pti_finalize(); system_state = SYSTEM_RUNNING; numa_default_policy(); rcu_end_inkernel_boot(); do_sysctl_args(); if (ramdisk_execute_command) { ret = run_init_process(ramdisk_execute_command); if (!ret) return 0; pr_err("Failed to execute %s (error %d)\n", ramdisk_execute_command, ret); } /* * We try each of these until one succeeds. * * The Bourne shell can be used instead of init if we are * trying to recover a really broken machine. */ if (execute_command) { ret = run_init_process(execute_command); if (!ret) return 0; panic("Requested init %s failed (error %d).", execute_command, ret); } if (CONFIG_DEFAULT_INIT[0] != '\0') { ret = run_init_process(CONFIG_DEFAULT_INIT); if (ret) pr_err("Default init %s failed (error %d)\n", CONFIG_DEFAULT_INIT, ret); else return 0; } if (!try_to_run_init_process("/sbin/init") || !try_to_run_init_process("/etc/init") || !try_to_run_init_process("/bin/init") || !try_to_run_init_process("/bin/sh")) return 0; panic("No working init found. Try passing init= option to kernel. " "See Linux Documentation/admin-guide/init.rst for guidance."); } ``` :::info kernel_init讀取Rootfs下的init( ("/sbin/init") 或 ("/etc/init") 或 ("/bin/init") )，PID為1。 ::: --- # Environment Host: Ubuntu x64 Board: Raspberry pi 4b JTAG: FT2232H-56Q(FT2232H Mini Module) [FTDI DATASHEET](https://ftdichip.com/wp-content/uploads/2020/07/DS_FT2232H_Mini_Module.pdf) Linux kernel [Raspberry pi linux kernel GITHUB](https://github.com/raspberrypi/linux) UBOOT [uboot GITHUB](https://github.com/u-boot/u-boot) Toolchain: gcc-aarch64-linux-gnu GDB: gdb-multiarch OpenOCD [OpenOCD GITHUB](https://github.com/openocd-org/openocd) Screen Console 工具 杜邦線 :::info 安裝toolchain ```bash= sudo apt install -y gcc-aarch64-linux-gnu ``` 安裝GDB ```bash= sudo apt install -y gdb-multiarch ``` 安裝Screen ```bash= sudo apt install -y screen ``` ::: :::info OpenOCD不要使用APT套件庫的，請使用GITHUB上的版本。APT套件庫的版本過舊。 編譯 ```bash= ./configure --enable-ftdi make -j6 sudo make install ``` ::: --- # KGDB Trace 流程: 1. Raspberry pi上電 2. 透過Console操作，讓他停在Uboot 3. HOST端開啟OpenOCD，啟動GDB Server 4. HOST端啟動GDB Client連接OpenOCD 5. GDB Client設定中斷點，並讓Raspberry pi繼續開機 6. Raspberry pi會停在中斷點。 :::info 一共會用到三個Terminator。 * Raspberry pi Console * OpenOCD * GDB Client ::: --- # Linux kernel 和 Uboot compile 請參考我之前Raspberry pi 4 Build System文章，這裡我就不多概述了。 [參考 Raspberry pi 4 Build System (64bits)](https://hackmd.io/p_eQ-5QdRE-NpweHBE4_Eg?view) 比較需要注意的有兩個地方。 1. 過去在Compile linux kernel 並沒有將KGDB相關功能加入Linux kernel，因此需要加入KGDB設定。 2. 將Raspberry pi開機參數寫入config.txt ### Linux kernel 加入KGDB設定 請依照以下步驟將KGDB設定加入設定檔後再編譯。 1. 在Linux kernel底下找到Raspberry pi的設定檔案 ```bash= cd ${linux_kernel 目錄} find . -name bcm2711_defconfig ``` 會出現以下兩行。第一行給32位元(ARM)用的設定檔，本文章使用第二個給64位元(ARM64)的設定檔案。 ```bash= ./arch/arm/configs/bcm2711_defconfig ./arch/arm64/configs/bcm2711_defconfig ``` 2. 在檔案之中(./arch/arm64/configs/bcm2711_defconfig)加入以下設定 ```text= CONFIG_DEBUG_INFO=y CONFIG_GDB_SCRIPTS=y CONFIG_FRAME_POINTER=y ``` 3. 設定，並編譯 ```bash= make -j6 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- bcm2711_defconfig make -j6 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- all ``` :::info GDB會用到原始為壓縮的Linux kernel -- vmlinux，所以編譯選擇all，將所有東西都編譯。 ::: ### 將Raspberry pi開機參數寫入config.txt * config.txt ```text= kernel=u-boot.bin arm_64bit=1 enable_uart=1 # Disable pull downs gpio=22-27=np # Enable jtag pins (i.e. GPIO22-GPIO27) enable_jtag_gpio=1 ``` 將設定檔放到Raspberry pi第一分區(FAT32開機磁區) --- # JTAG on raspberry pi 要說到Raspberry pi的JTAG Pin位置，首先我們得先知道Raspberry pi使用的SOC(ARM architecture)JTAG Pin，是哪幾隻。 如下圖，ARM JTAG主要使用TMS(測試模式狀態PIN)，TDO(輸出)，TDI(輸入)，TCLK(時脈)傳輸訊號，VCC，GND供電，nRESET重置腳位。[ARM Cortex A JTAG interface](https://www.keil.com/support/man/docs/ulinkpro/ulinkpro_hw_if_jtag10.htm) ![](https://i.imgur.com/29r4izl.png) 接著在Raspberry PI上找到一模一樣功能的腳位(GPIO22 ~ GPIO27)。 [Raspberry pi 4b SPEC](https://datasheets.raspberrypi.org/rpi4/raspberry-pi-4-datasheet.pdf) ![](https://i.imgur.com/Q7pWZyZ.png) 實際將GPIO對應到PIN header[參考 COSCUP 2021 KGDB conference](https://hackmd.io/@coscup/Bk8qLaw0d/%2F%40coscup%2FrJu8I6PCO)。 ![](https://i.imgur.com/7uwYavu.png) [參考 JTAG - Joint Test Action Group](https://pinout.xyz/pinout/jtag#) ![](https://i.imgur.com/4SFyQoZ.png) --- # FT2232H JTAG FT2232H 是一個由FTDI設計，USB對多用途的IC。支援功能如下[SPEC](https://ftdichip.com/wp-content/uploads/2020/07/DS_FT2232H_Mini_Module.pdf): ```text= Single chip USB to dual channel UART (RS232, RS422 or RS485). * Single chip USB to dual channel FIFO. * Single chip USB to dual channel JTAG. * Single chip USB to dual channel SPI. * Single chip USB to dual channel I2C. * Single chip USB to dual channel Bit-Bang. * Single chip USB to dual combination of any of above interfaces. * Single chip USB to Fast Serial Optic Interface. * Single chip USB to CPU target interface (as memory), double and independent. * Single chip USB to Host Bus Emulation (as CPU). * PDA to USB data transfer * USB Smart Card Readers * USB Instrumentation * USB Industrial Control * USB MP3 Player Interface * USB FLASH Card Reader / Writers * Set Top Box PC - USB interface * USB Digital Camera Interface * USB Bar Code Readers ``` 腳位定義如下，可以看出GPIO16~GPIO19，是JTAG的Pin。 ![](https://i.imgur.com/awpDjge.png) --- # FT2232H 與 Raspberry pi 接線 [參考 Baremetal Raspberry Pi 4 with FT2232H](https://www.vinnie.work/blog/2021-04-02-ft2232h-rpi4/) ![](https://i.imgur.com/un8QKrb.png) :::info 為了操作Raspberry pi，有將兩者UART對接，以便可透過Console操作Raspberry pi。 ::: --- # 設定OpenOCD 1. 建立OpenOCD設定檔案(minimodule.cfg和raspi4.cfg) * minimodule.cfg: ```text= # # FTDI MiniModule # # http://www.ftdichip.com/Support/Documents/DataSheets/Modules/DS_FT2232H_Mini_Module.pdf # echo "WARNING!" echo "This file was not tested with real interface, it is based on code in ft2232.c." echo "Please report your experience with this file to openocd-devel mailing list," echo "so it could be marked as working or fixed." adapter driver ftdi ftdi_device_desc "FT2232H MiniModule" ftdi_vid_pid 0x0403 0x6010 ftdi_layout_init 0x0018 0x05fb ftdi_layout_signal nSRST -data 0x0020 ``` * raspi4.cfg: ```text= set _CHIPNAME bcm2711 set _DAP_TAPID 0x4ba00477 adapter_khz 1000 transport select jtag reset_config trst_and_srst telnet_port 4444 # create tap jtag newtap auto0 tap -irlen 4 -expected-id $_DAP_TAPID # create dap dap create auto0.dap -chain-position auto0.tap set CTIBASE {0x80420000 0x80520000 0x80620000 0x80720000} set DBGBASE {0x80410000 0x80510000 0x80610000 0x80710000} set _cores 4 set _TARGETNAME $_CHIPNAME.a72 set _CTINAME $_CHIPNAME.cti set _smp_command "" for {set _core 0} {$_core < $_cores} { incr _core} { cti create $_CTINAME.$_core -dap auto0.dap -ap-num 0 -ctibase [lindex $CTIBASE $_core] set _command "target create ${_TARGETNAME}.$_core aarch64 \ -dap auto0.dap -dbgbase [lindex $DBGBASE $_core] \ -coreid $_core -cti $_CTINAME.$_core" if {$_core != 0} { set _smp_command "$_smp_command $_TARGETNAME.$_core" } else { set _smp_command "target smp $_TARGETNAME.$_core" } eval $_command } eval $_smp_command targets $_TARGETNAME.0 ``` # 開始Trace囉 #### 開啟Raspberry pi Console 1. 先將FT2232H接上電腦，並開啟Console: 會出現兩個USB Interface，選擇第二個 ![](https://i.imgur.com/U6jxOId.png) ```bash= sudo screen /dev/ttyUSB1 115200 ``` 2. Raspberry PI上電後，持續按下鍵盤上的Enter(持續點放，不是持續按著): 直到進入Uboot命令列，如下圖 ![](https://i.imgur.com/g201h5J.png) #### 開啟OpenOCD 和 GDB server 開啟另一個終端機，輸入以下命令(記得先切到minimodule.cfg，raspi4.cfg存放的目錄): ```bash= sudo openocd -f minimodule.cfg -c "set USE_SMP 1" -f raspi4.cfg -c "reset_config trst_only" ``` ![](https://i.imgur.com/nYRQU6j.png) * 以上步驟我們將Console，OpenOCD，GDB Server準備好了。 #### 開啟OpenOCD 和 GDB client 1. 開啟另一個終端機，切換到剛剛Build Linux kernel的目錄，並輸入: ```bash= gdb-multiarch -q vmlinux ``` ![](https://i.imgur.com/kdnl3yE.png) 2. 連接GDB Server: ```bash= target extended-remote :3333 ``` ![](https://i.imgur.com/PJgRsw6.png) :::info 這時會發現Raspberry pi的Console停住了，無法輸入文字或是控制。這是因為Raspberry pi被OpenOCD freeze住了。 ::: 3. 下中斷點: ```bash= hbreak kernel_init ``` ![](https://i.imgur.com/O1szARB.png) 4. 讓Raspberry PI繼續執行: ```bash= c ``` ![](https://i.imgur.com/ACmTDDC.png) :::info c (continue)，解除freeze ::: #### Raspberry PI console: 輸入開機參數，載入Linux kernel，並開機: ```bash= setenv bootargs 8250.nr_uarts=1 console=ttyS0,115200 root=/dev/mmcblk0p2 rootwait rw fatload mmc 0:1 ${kernel_addr_r} Image booti ${kernel_addr_r} - ${fdt_addr} ``` ![](https://i.imgur.com/lBopN99.png) #### 發現GDB Client停在我們下的中斷點: ![](https://i.imgur.com/2qSii8a.png) * 使用backtrace可以追蹤function caller: ![](https://i.imgur.com/C4R4xT2.png) :::info 到這裡可以使用GDB指令去追蹤kernel_init了 ::: :::info GDB詳細使用方式可參考這篇文章[透過 GDB 進行遠端除錯](https://hackmd.io/@sysprog/gdb-example) :::