--- hackmd: url: https://hackmd.io/3q1h2ledS0id4mipdI4p7w title: Background lastSync: 2025-04-24T05:28:27.316Z --- Yocto Project gdb cross debug ===== # Background - When working with embedded systems like BMCs, limited memory often means that full debugging tools like GDB aren’t available on the target image. Fortunately, there are two practical approaches for debugging in a Yocto-based environment: using gdbserver for live debugging, or analyzing core dump files offline. This article walks you through how to set up both. - 當程式有錯誤的時候可以使用兩種方式去除錯，使用gdbserver進行cross debugging 或是使用產生的coredump file進行除錯 # Example - Let's start with a very basic example. We have a service called hello-recipe, which encounters a segmentation fault due to dereferencing a null pointer. - 假設有一個service叫做hello-recipe，其main.cpp有Dereferencing a null pointer 造成的 segmentation fault: ```cpp= // main.cpp #include <iostream> int main() { std::cout << "Hello, World!" << std::endl; int* ptr = nullptr; std::cout << *ptr << std::endl; // Dereferencing a null pointer causes a segmentation fault (core dump) return 0; } ``` - we can checkout log using systemctl - 檢查service ```shell root@ums120:~# systemctl status hello-recipe x hello-recipe.service - Hello Recipe Service Loaded: loaded (/usr/lib/systemd/system/hello-recipe.service; enabled; preset: enabled) Active: failed (Result: core-dump) since Thu 2025-04-24 02:13:20 UTC; 32min ago Duration: 2.505s Process: 490 ExecStart=/usr/bin/hello (code=dumped, signal=SEGV) Main PID: 490 (code=dumped, signal=SEGV) CPU: 106ms Apr 24 02:13:20 ums120 systemd[1]: hello-recipe.service: Scheduled restart job, restart counter is at 2. Apr 24 02:13:20 ums120 systemd[1]: hello-recipe.service: Start request repeated too quickly. Apr 24 02:13:20 ums120 systemd[1]: hello-recipe.service: Failed with result 'core-dump'. Apr 24 02:13:20 ums120 systemd[1]: Failed to start Hello Recipe Service. ``` - Yocto Setting - build/conf/local.conf ``` IMAGE_INSTALL:append = " coreutils" EXTRA_OECONF += "ULIMIT_CORE=unlimited" CONF_VERSION = "1" EXTRA_OECONF += "KERNEL_PARAM_CORE_PATTERN=/var/lib/systemd/coredump/core.%e.%p.%t" IMAGE_INSTALL:append = " gdbserver" ``` - Serivce Setting - demo-hello/hello-recipe.bb ``` # 保留 DWARF、禁 strip INHIBIT_PACKAGE_STRIP        = "1" INHIBIT_PACKAGE_DEBUG_SPLIT  = "0" do_compile() {     oe_runmake \         CXXFLAGS="${CXXFLAGS} -O0 -g3 -fno-omit-frame-pointer -fno-PIE" \         LDFLAGS="${LDFLAGS} -no-pie" } ``` - build image - `bitbake obmc-phosphor-image` - generate sdk - `bitbake -c populate_sdk obmc-phosphor-image` - download sdk - `/build/tmp/deploy/sdk$ ./oecore-obmc-phosphor-image-x86_64-armv7ahf-vfpv4d16-ums120-toolchain-nodistro.0.sh` ``` Phosphor OpenBMC (Phosphor OpenBMC Project Reference Distro) SDK installer version nodistro.0 ============================================================================================= Enter target directory for SDK (default: /usr/local/oecore-x86_64): The directory "/usr/local/oecore-x86_64" already contains a SDK for this architecture. If you continue, existing files will be overwritten! Proceed [y/N]? y [sudo] password for alan: Extracting SDK.........................................................................................................................................done Setting it up...done SDK has been successfully set up and is ready to be used. Each time you wish to use the SDK in a new shell session, you need to source the environment setup script e.g. $ . /usr/local/oecore-x86_64/environment-setup-armv7ahf-vfpv4d16-openbmc-linux-gnueabi ``` - setup environment - `. /usr/local/oecore-x86_64/environment-setup-armv7ahf-vfpv4d16-openbmc-linux-gnueabi` # Cross Debugging ## option 1: remote debug - In remote embedded chips e.g. bmc - open gdbserver - gdbserver :1234 /usr/bin/hello - enter gdb - `build/tmp/deploy/sdk$ arm-openbmc-linux-gnueabi-gdb` - setup sysroot and code entry ``` (gdb) set sysroot /usr/local/oecore-x86_64/sysroots/armv7ahf-vfpv4d16-openbmc-linux-gnueabi (gdb) set substitute-path /usr/src/debug/hello-recipe/1.0 /home/alan/beacon_ums120_cb/meta-wiwynn/meta-ums120/recipes-phosphor/demo-hello/files ``` - connect to remote `target remote $REMOTE_IP:1234` ```shell= (gdb) target remote 10.248.42.110:1234 Remote debugging using 10.248.42.110:1234 Reading symbols from /usr/local/oecore-x86_64/sysroots/armv7ahf-vfpv4d16-openbmc-linux-gnueabi/usr/bin/hello... Reading symbols from /usr/local/oecore-x86_64/sysroots/armv7ahf-vfpv4d16-openbmc-linux-gnueabi/usr/lib/ld-linux-armhf.so.3... Reading symbols from /usr/local/oecore-x86_64/sysroots/armv7ahf-vfpv4d16-openbmc-linux-gnueabi/usr/lib/.debug/ld-linux-armhf.so.3... 0x76fee838 in _start () from /usr/local/oecore-x86_64/sysroots/armv7ahf-vfpv4d16-openbmc-linux-gnueabi/usr/lib/ld-linux-armhf.so.3 ``` - setup break point ``` (gdb) break main Breakpoint 1 at 0x106d4: file main.cpp, line 5. ``` - debug step by step 單步執行除錯 ```cpp= (gdb) continue Continuing. Breakpoint 1, main () at main.cpp:5 5 std::cout << "Hello, World!" << std::endl; (gdb) continue Continuing. Program received signal SIGSEGV, Segmentation fault. 0x00010708 in main () at main.cpp:7 7 std::cout << *ptr << std::endl; // Dereferencing a null pointer causes a segmentation fault (core dump) ``` ## option 2 using coredump coredump - at earlier systemctl log, we can know that process 490 has problem. 從systemctl可知是 process 490有錯 - find coredump file in remote 在遠端裝置查看core file 位置 ```shell= root@ums120:~# ls /var/lib/systemd/coredump/ core.hello.0.26b9a60af04d45d1a37c84d334a7c353.261.1745460784000000.zst core.hello.0.26b9a60af04d45d1a37c84d334a7c353.490.1745460796000000.zst ``` - download binary and core file to host. 下載binary及core file到host除錯 ```shell= scp root@10.248.42.110:/usr/bin/hello hello.debug scp root@10.248.42.110:/var/lib/systemd/coredump/core.hello.0.26b9a60af04d45d1a37c84d334a7c353.490.1745460796000000.zst . ``` - decompress core file 解壓縮core file ```shell $ zstd -d core.hello.0.26b9a60af04d45d1a37c84d334a7c353.490.1745460796000000.zst -o hello.core $ scp root@192.168.5.6:/usr/bin/hello hello.debug ``` - using gdb to debug 用gdb進行除錯 ```shell= arm-openbmc-linux-gnueabi-gdb hello.debug hello.core GNU gdb (GDB) 14.2 Copyright (C) 2023 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "--host=x86_64-oesdk-linux --target=arm-openbmc-linux-gnueabi". Type "show configuration" for configuration details. For bug reporting instructions, please see: <https://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from hello.debug... ``` - setup sysroot and code entry 設定 sysroot 及 code位置 ```shell= (gdb) set sysroot /usr/local/oecore-x86_64/sysroots/armv7ahf-vfpv4d16-openbmc-linux-gnueabi Reading symbols from /usr/local/oecore-x86_64/sysroots/armv7ahf-vfpv4d16-openbmc-linux-gnueabi/lib/libstdc++.so.6... Reading symbols from /usr/local/oecore-x86_64/sysroots/armv7ahf-vfpv4d16-openbmc-linux-gnueabi/lib/.debug/libstdc++.so.6.0.32... Reading symbols from /usr/local/oecore-x86_64/sysroots/armv7ahf-vfpv4d16-openbmc-linux-gnueabi/lib/libgcc_s.so.1... Reading symbols from /usr/local/oecore-x86_64/sysroots/armv7ahf-vfpv4d16-openbmc-linux-gnueabi/lib/.debug/libgcc_s.so.1... Reading symbols from /usr/local/oecore-x86_64/sysroots/armv7ahf-vfpv4d16-openbmc-linux-gnueabi/usr/lib/libc.so.6... Reading symbols from /usr/local/oecore-x86_64/sysroots/armv7ahf-vfpv4d16-openbmc-linux-gnueabi/usr/lib/.debug/libc.so.6... Reading symbols from /usr/local/oecore-x86_64/sysroots/armv7ahf-vfpv4d16-openbmc-linux-gnueabi/lib/libm.so.6... Reading symbols from /usr/local/oecore-x86_64/sysroots/armv7ahf-vfpv4d16-openbmc-linux-gnueabi/lib/.debug/libm.so.6... Reading symbols from /usr/local/oecore-x86_64/sysroots/armv7ahf-vfpv4d16-openbmc-linux-gnueabi/usr/lib/ld-linux-armhf.so.3... Reading symbols from /usr/local/oecore-x86_64/sysroots/armv7ahf-vfpv4d16-openbmc-linux-gnueabi/usr/lib/.debug/ld-linux-armhf.so.3... (gdb) set substitute-path /usr/src/debug/hello-recipe/1.0 /home/alan/beacon_ums120_cb/meta-wiwynn/meta-ums120/recipes-phosphor/demo-hello/files ``` - backtrace ```shell (gdb) backtrace #0 0x00010708 in main () at main.cpp:7 (gdb) frame 0 #0 0x00010708 in main () at main.cpp:7 7 std::cout << *ptr << std::endl; // Dereferencing a null pointer causes a segmentation fault (core dump) (gdb) list 2 #include <iostream> 3 4 int main() { 5 std::cout << "Hello, World!" << std::endl; 6 int* ptr = nullptr; 7 std::cout << *ptr << std::endl; // Dereferencing a null pointer causes a segmentation fault (core dump) 8 return 0; 9 } ```