Try   HackMD

Yocto Project gdb cross debug

Background

  • When working with embedded systems like BMCs, limited memory often means that full debugging tools like GDB aren’t available on the target image. Fortunately, there are two practical approaches for debugging in a Yocto-based environment: using gdbserver for live debugging, or analyzing core dump files offline. This article walks you through how to set up both.
  • 當程式有錯誤的時候可以使用兩種方式去除錯，使用gdbserver進行cross debugging 或是使用產生的coredump file進行除錯

Example

  • Let's start with a very basic example. We have a service called hello-recipe, which encounters a segmentation fault due to dereferencing a null pointer.
  • 假設有一個service叫做hello-recipe，其main.cpp有Dereferencing a null pointer 造成的 segmentation fault:









// main.cpp
#include <iostream>

int main() {
	std::cout << "Hello, World!" << std::endl;
	int* ptr = nullptr;
	std::cout << *ptr << std::endl; // Dereferencing a null pointer causes a segmentation fault (core dump)
	return 0;
}
  • we can checkout log using systemctl
  • 檢查service
root@ums120:~# systemctl status hello-recipe
x hello-recipe.service - Hello Recipe Service
     Loaded: loaded (/usr/lib/systemd/system/hello-recipe.service; enabled; preset: enabled)
     Active: failed (Result: core-dump) since Thu 2025-04-24 02:13:20 UTC; 32min ago
   Duration: 2.505s
    Process: 490 ExecStart=/usr/bin/hello (code=dumped, signal=SEGV)
   Main PID: 490 (code=dumped, signal=SEGV)
        CPU: 106ms

Apr 24 02:13:20 ums120 systemd[1]: hello-recipe.service: Scheduled restart job, restart counter is at 2.
Apr 24 02:13:20 ums120 systemd[1]: hello-recipe.service: Start request repeated too quickly.
Apr 24 02:13:20 ums120 systemd[1]: hello-recipe.service: Failed with result 'core-dump'.
Apr 24 02:13:20 ums120 systemd[1]: Failed to start Hello Recipe Service.
  • Yocto Setting
    • build/conf/local.conf
IMAGE_INSTALL:append = " coreutils"

EXTRA_OECONF += "ULIMIT_CORE=unlimited"

CONF_VERSION = "1"

EXTRA_OECONF += "KERNEL_PARAM_CORE_PATTERN=/var/lib/systemd/coredump/core.%e.%p.%t"

IMAGE_INSTALL:append = " gdbserver"
  • Serivce Setting
    • demo-hello/hello-recipe.bb
# 保留 DWARF、禁 strip
INHIBIT_PACKAGE_STRIP        = "1"
INHIBIT_PACKAGE_DEBUG_SPLIT  = "0"
do_compile() {
    oe_runmake \
        CXXFLAGS="${CXXFLAGS} -O0 -g3 -fno-omit-frame-pointer -fno-PIE" \
        LDFLAGS="${LDFLAGS} -no-pie"
}
  • build image
    • bitbake obmc-phosphor-image
  • generate sdk
    • bitbake -c populate_sdk obmc-phosphor-image
  • download sdk
    • /build/tmp/deploy/sdk$ ./oecore-obmc-phosphor-image-x86_64-armv7ahf-vfpv4d16-ums120-toolchain-nodistro.0.sh
Phosphor OpenBMC (Phosphor OpenBMC Project Reference Distro) SDK installer version nodistro.0
=============================================================================================
Enter target directory for SDK (default: /usr/local/oecore-x86_64):
The directory "/usr/local/oecore-x86_64" already contains a SDK for this architecture.
If you continue, existing files will be overwritten! Proceed [y/N]? y
[sudo] password for alan:
Extracting SDK.........................................................................................................................................done
Setting it up...done
SDK has been successfully set up and is ready to be used.
Each time you wish to use the SDK in a new shell session, you need to source the environment setup script e.g.
 $ . /usr/local/oecore-x86_64/environment-setup-armv7ahf-vfpv4d16-openbmc-linux-gnueabi
  • setup environment
    • . /usr/local/oecore-x86_64/environment-setup-armv7ahf-vfpv4d16-openbmc-linux-gnueabi

Cross Debugging

option 1: remote debug

  • In remote embedded chips e.g. bmc
    • open gdbserver
      • gdbserver :1234 /usr/bin/hello
    • enter gdb
      • build/tmp/deploy/sdk$ arm-openbmc-linux-gnueabi-gdb
    • setup sysroot and code entry
    ​(gdb) set sysroot /usr/local/oecore-x86_64/sysroots/armv7ahf-vfpv4d16-openbmc-linux-gnueabi
​(gdb) set substitute-path /usr/src/debug/hello-recipe/1.0 /home/alan/beacon_ums120_cb/meta-wiwynn/meta-ums120/recipes-phosphor/demo-hello/files
    • connect to remote target remote $REMOTE_IP:1234
    





    ​	(gdb) target remote 10.248.42.110:1234
​	Remote debugging using 10.248.42.110:1234
​	Reading symbols from /usr/local/oecore-x86_64/sysroots/armv7ahf-vfpv4d16-openbmc-linux-gnueabi/usr/bin/hello...
​	Reading symbols from /usr/local/oecore-x86_64/sysroots/armv7ahf-vfpv4d16-openbmc-linux-gnueabi/usr/lib/ld-linux-armhf.so.3...
​	Reading symbols from /usr/local/oecore-x86_64/sysroots/armv7ahf-vfpv4d16-openbmc-linux-gnueabi/usr/lib/.debug/ld-linux-armhf.so.3...
​	0x76fee838 in _start () from /usr/local/oecore-x86_64/sysroots/armv7ahf-vfpv4d16-openbmc-linux-gnueabi/usr/lib/ld-linux-armhf.so.3
    • setup break point
(gdb) break main
Breakpoint 1 at 0x106d4: file main.cpp, line 5.
  • debug step by step 單步執行除錯












(gdb) continue
Continuing.

Breakpoint 1, main () at main.cpp:5
5           std::cout << "Hello, World!" << std::endl;
(gdb) continue
Continuing.

Program received signal SIGSEGV, Segmentation fault.
0x00010708 in main () at main.cpp:7
7           std::cout << *ptr << std::endl; // Dereferencing a null pointer causes a segmentation fault (core dump)

option 2 using coredump coredump

  • at earlier systemctl log, we can know that process 490 has problem. 從systemctl可知是 process 490有錯
  • find coredump file in remote 在遠端裝置查看core file 位置



root@ums120:~# ls /var/lib/systemd/coredump/
core.hello.0.26b9a60af04d45d1a37c84d334a7c353.261.1745460784000000.zst
core.hello.0.26b9a60af04d45d1a37c84d334a7c353.490.1745460796000000.zst
  • download binary and core file to host. 下載binary及core file到host除錯


scp root@10.248.42.110:/usr/bin/hello hello.debug 
scp root@10.248.42.110:/var/lib/systemd/coredump/core.hello.0.26b9a60af04d45d1a37c84d334a7c353.490.1745460796000000.zst .
  • decompress core file 解壓縮core file
zstd -d core.hello.0.26b9a60af04d45d1a37c84d334a7c353.490.1745460796000000.zst -o hello.core
  • using gdb to debug 用gdb進行除錯

















arm-openbmc-linux-gnueabi-gdb hello.debug hello.core
GNU gdb (GDB) 14.2
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "--host=x86_64-oesdk-linux --target=arm-openbmc-linux-gnueabi".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from hello.debug...
  • setup sysroot and code entry 設定 sysroot 及 code位置













	(gdb) set sysroot /usr/local/oecore-x86_64/sysroots/armv7ahf-vfpv4d16-openbmc-linux-gnueabi
	Reading symbols from /usr/local/oecore-x86_64/sysroots/armv7ahf-vfpv4d16-openbmc-linux-gnueabi/lib/libstdc++.so.6...
Reading symbols from /usr/local/oecore-x86_64/sysroots/armv7ahf-vfpv4d16-openbmc-linux-gnueabi/lib/.debug/libstdc++.so.6.0.32...
Reading symbols from /usr/local/oecore-x86_64/sysroots/armv7ahf-vfpv4d16-openbmc-linux-gnueabi/lib/libgcc_s.so.1...
Reading symbols from /usr/local/oecore-x86_64/sysroots/armv7ahf-vfpv4d16-openbmc-linux-gnueabi/lib/.debug/libgcc_s.so.1...
Reading symbols from /usr/local/oecore-x86_64/sysroots/armv7ahf-vfpv4d16-openbmc-linux-gnueabi/usr/lib/libc.so.6...
Reading symbols from /usr/local/oecore-x86_64/sysroots/armv7ahf-vfpv4d16-openbmc-linux-gnueabi/usr/lib/.debug/libc.so.6...
Reading symbols from /usr/local/oecore-x86_64/sysroots/armv7ahf-vfpv4d16-openbmc-linux-gnueabi/lib/libm.so.6...
Reading symbols from /usr/local/oecore-x86_64/sysroots/armv7ahf-vfpv4d16-openbmc-linux-gnueabi/lib/.debug/libm.so.6...
Reading symbols from /usr/local/oecore-x86_64/sysroots/armv7ahf-vfpv4d16-openbmc-linux-gnueabi/usr/lib/ld-linux-armhf.so.3...
Reading symbols from /usr/local/oecore-x86_64/sysroots/armv7ahf-vfpv4d16-openbmc-linux-gnueabi/usr/lib/.debug/ld-linux-armhf.so.3...

	(gdb) set substitute-path /usr/src/debug/hello-recipe/1.0 /home/alan/beacon_ums120_cb/meta-wiwynn/meta-ums120/recipes-phosphor/demo-hello/files
  • backtrace
(gdb) backtrace
#0  0x00010708 in main () at main.cpp:7
(gdb) frame 0
#0  0x00010708 in main () at main.cpp:7
7           std::cout << *ptr << std::endl; // Dereferencing a null pointer causes a segmentation fault (core dump)
(gdb) list
2       #include <iostream>
3
4       int main() {
5           std::cout << "Hello, World!" << std::endl;
6           int* ptr = nullptr;
7           std::cout << *ptr << std::endl; // Dereferencing a null pointer causes a segmentation fault (core dump)
8           return 0;
9       }
Last changed by 
 
alanhc
https://www.linkedin.com/in/alanhc316/
1
574

Read more

til

index

Jun 19, 2025
profiling 筆記

效能可以被分解為最根本的三個資源維度：

Jun 17, 2025
memory

huge page:→一種比標準 4KB 更大的記憶體頁面（如 2MB 或 1GB），減少 TLB misses，提高效能。

Jun 17, 2025
t-mac

paper: T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge

Jun 17, 2025
Read more from alanhc
Published on HackMD