# Debug Clang/LLVM Compiled Arm64 Linux Kernel on QEMU Virt with GDB
[toc]
## Enviroment Setup
### 1. Initramfs, Linux Kernel, external modules and QEMU
Please see [Run Clang/LLVM Compiled Arm64 Linux Kernel on QEMU Virt with BusyBox Initramfs](https://hackmd.io/@Ming-Jun/SJTBZOA0A)
### 2. GDB
1. Install `gdb` toolchain and check version
```shell
$ sudo apt-get install gdb-multiarch
$ gdb-multiarch --version
GNU gdb (Ubuntu 15.0.50.20240403-0ubuntu1) 15.0.50.20240403-git
```
2. Create `${HOME}/.config/gdb/gdbinit` with the following content
```shell
add-auto-load-safe-path /your/path/to/linux/scripts/gdb/vmlinux-gdb.py
set auto-load safe-path /
```
## Debug Linux Kernel with GDB
### 1. Exam `vmlinux` via `readelf`
* Producer: `DW_AT_producer`
This debug information provides the final build command.
```shell
$ readelf -Wwi vmlinux | grep "DW_AT_producer"
```
* Compilation directory: `DW_AT_comp_dir`
GDB will use this path to search source code. If the path is incorrect, you can use `directory` or `substitute-path` to let GDB know where to search. Visit ["Specifying Source Directories"](https://sourceware.org/gdb/current/onlinedocs/gdb.html/Source-Path.html) for more information.
```shell
$ readelf -Wwi vmlinux | grep "DW_AT_comp_dir"
DW_AT_comp_dir : (string) /your/path/to/linux
```
### 2. Run QEMU
Add GDB option `-gdb tcp::1234`, CPU option `-S` and Kernel bootargs `nokaslr`, `arm64.nopauth` and `cpuidle.off=1`.
```shell
$ qemu-system-aarch64 \
-machine virt \
-m size=1024M \
-cpu cortex-a57 \
-smp 2 \
-nographic \
-initrd /your/path/to/busybox-1.36.1/initramfs.cpio.gz \
-kernel /your/path/to/linux/arch/arm64/boot/Image \
--append "console=ttyAMA0 nokaslr arm64.nopauth cpuidle.off=1" \
-gdb tcp::1234 \
-S
```
### 3. Run GDB
#### In a terminal
1. Run `gdb-multiarch` with `vmlinux` and then attach to the remote target on `localhost: 1234`
```shell
$ gdb-multiarch /your/path/to/linux/vmlinux -ex 'target remote localhost: 1234'
```
2. Check GDB source directories
The `$cdir` means compilation directory. GDB will get `$cdir` from the debug information `DW_AT_comp_dir` in `vmlinux`
```shell
(gdb) show directories
Source directories searched: $cdir:$cwd
```
3. Set a breakpoint at the `start_kernel` function
```shell
(gdb) b start_kernel
Breakpoint 1 at 0xffff800081ad03d4: file init/main.c, line 875.
```
4. Continue and wait for breakpoints to be triggered
```shell
(gdb) c
Continuing.
Thread 1 hit Breakpoint 1, start_kernel () at init/main.c:875
875 set_task_stack_end_magic(&init_task);
```
5. Run `list .` command to show source code
```shell
(gdb) list .
870 void start_kernel(void)
871 {
872 char *command_line;
873 char *after_dashes;
874
875 set_task_stack_end_magic(&init_task);
876 smp_setup_processor_id();
877 debug_objects_early_init();
878 init_vmlinux_build_id();
879
```
6. Run `tui enable` command for better user experience
```shell
(gdb) tui enable
```
7. Start debugging!
#### In VS code
1. Install VS code [Native Debug](https://marketplace.visualstudio.com/items?itemName=webfreak.debug) extension
2. Open Linux Kernel source code root directory with VS code
```shell
$ code /your/path/to/linux
```
3. Create `.vscode/launch.json` with the following content
```json
{
"version": "0.2.0",
"configurations": [
{
"type": "gdb",
"request": "attach",
"name": "Attach to gdbserver",
"gdbpath":"/your/path/to/gdb-multiarch",
"executable": "/your/path/to/vmlinux",
"debugger_args" : [],
"target": "localhost:1234",
"remote": true,
"cwd": "${workspaceRoot}",
"valuesFormatting": "parseText"
}
]
}
```
4. Set a breakpoint for the `start_kernel` function at file `init/main.c` line 875
5. Run debugger and wait for breakpoints to be triggered
6. Start debugging!
## Debug External Modules with GDB
### 1. Exam `exampme_module.ko` via `readelf` and `llvm-objdump`
* Producer: `DW_AT_producer`
This debug information provides the final build command.
```shell
$ readelf -Wwi example_module.ko | grep "DW_AT_producer"
```
* Compilation directory: `DW_AT_comp_dir`
GDB will use this path to search source code. If the path is incorrect, you can use `directory` or `substitute-path` to let GDB know where to search. Visit ["Specifying Source Directories"](https://sourceware.org/gdb/current/onlinedocs/gdb.html/Source-Path.html) for more information.
```shell
$ readelf -Wwi example_module.ko | grep "DW_AT_comp_dir"
<19> DW_AT_comp_dir : (strx1) (offset: 0x2): /your/path/to/linux
```
You might notice that the compilation directory is the Linux Kernel source directory instead of the externel modules source directory. So we need to manually add the externel modules source directory to GDB via `directory` later.
* List `.text` sections
```shell
$ readelf -WS example_module.ko | grep "text"
Section Headers:
[Nr] Name Type Address Off Size ES Flg Lk Inf Al
[ 3] .text.ftrace_trampoline PROGBITS 0000000000000000 000042 000001 00 A 0 0 1
[ 4] .text PROGBITS 0000000000000000 000044 000000 00 AX 0 0 4
[ 5] .init.text PROGBITS 0000000000000000 000044 000028 00 AX 0 0 4
[ 6] .rela.init.text RELA 0000000000000000 000648 000048 18 I 37 5 8
[ 7] .exit.text PROGBITS 0000000000000000 00006c 000024 00 AX 0 0 4
[ 8] .rela.exit.text RELA 0000000000000000 000690 000048 18 I 37 7 8
```
* Check the function you want to set breakpoint at is in which section. Here we can see `hellowolrd_init` is in `.init.text` section.
```shell
$ llvm-objdump -d -ClS example_module.ko
example_module.ko: file format elf64-littleaarch64
Disassembly of section .init.text:
0000000000000000 <init_module>:
; __this_module():
; /your/path/to/external_modules/example_module/example_module.c:5
; static int __init hellowolrd_init(void) {
0: d503233f paciasp
4: a9bf7bfd stp x29, x30, [sp, #-0x10]!
8: 910003fd mov x29, sp
; /your/path/to/external_modules/example_module/example_module.c:6
; pr_info("Hello world!\n");
c: 90000000 adrp x0, 0x0 <init_module>
10: 91000000 add x0, x0, #0x0
14: 94000000 bl 0x14 <init_module+0x14>
; _note_15():
; /your/path/to/external_modules/example_module/example_module.c:7
; return 0;
18: 2a1f03e0 mov w0, wzr
1c: a8c17bfd ldp x29, x30, [sp], #0x10
20: d50323bf autiasp
24: d65f03c0 ret
```
* Add more sections for GDB `lx-symbols` command
Currently, GDB `lx-symbols` command only support `.text`, `.data`, `.data..read_mostly`, `.rodata`, `.bss`, `.text.hot`, `.text.unlikely` sections. If you need more sections, please add them into the source code at `/your/path/to/linux/scripts/gdb/linux/symbols.py` line 106. For breakpoints in `hellowolrd_init` function, `.init.text` must be added into `symbols.py`
### 2. Run QEMU
Add GDB option `-gdb tcp::1234`, CPU option `-S` and Kernel bootargs `nokaslr`, `arm64.nopauth` and `cpuidle.off=1`.
```shell
$ qemu-system-aarch64 \
-machine virt \
-m size=1024M \
-cpu cortex-a57 \
-smp 2 \
-nographic \
-initrd /your/path/to/busybox-1.36.1/initramfs.cpio.gz \
-kernel /your/path/to/linux/arch/arm64/boot/Image \
--append "console=ttyAMA0 nokaslr arm64.nopauth cpuidle.off=1" \
-gdb tcp::1234 \
-S
```
### 3. Run GDB
#### In a terminal
1. Run `gdb-multiarch` with `vmlinux`
```shell
$ gdb-multiarch /your/path/to/linux/vmlinux
```
2. Add `your/path/to/external_modules/example_module` to GDB
```shell
(gdb) directory /your/path/to/external_modules/example_module
Source directories searched: /your/path/to/external_modules/example_module:$cdir:$cwd
```
3. Run `lx-symbols` command with `/your/path/to/external_modules/example_module`
```shell
(gdb) lx-symbols /your/path/to/external_modules/example_module
loading vmlinux
```
4. attach to the remote target on `localhost: 1234`
```shell
(gdb) target remote localhost: 1234
```
5. Set a breakpoint at the `hellowolrd_init` function and then continue
```shell
(gdb) b hellowolrd_init
Function "hellowolrd_init" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (hellowolrd_init) pending.
(gdb) c
Continuing.
```
6. Load `example_module` in QEMU
```shell
# modprobe example_module
[ 6.649931] example_module: loading out-of-tree module taints kernel.
```
7. Wait for breakpoints to be triggered
```shell
scanning for modules in /your/path/to/external_modules/example_module
loading @0xffff80007a1c0000: /your/path/to/external_modules/example_module/example_module.ko
[Switching to Thread 1.2]
Thread 2 hit Breakpoint 1, hellowolrd_init ()
at /your/path/to/external_modules/example_module/example_module.c:6
6 pr_info("Hello world!\n");
```
8. Run `info source` command to check the current source file
```shell
(gdb) info source
Current source file is /your/path/to/external_modules/example_module/example_module.c
Compilation directory is /your/path/to/linux
Located in /your/path/to/external_modules/example_module/example_module.c
Contains 17 lines.
Source language is c.
```
9. Run `list .` command to show source code
```shell
(gdb) list .
1 #include <linux/init.h>
2 #include <linux/module.h>
3 #include <linux/kernel.h>
4
5 static int __init hellowolrd_init(void) {
6 pr_info("Hello world!\n");
7 return 0;
8 }
9
10 static void __exit hellowolrd_exit(void) {
```
10. Run `tui enable` command for better user experience
```shell
(gdb) tui enable
```
11. Start debugging!
#### In VS code
1. Install VS code [Native Debug](https://marketplace.visualstudio.com/items?itemName=webfreak.debug) extension
2. Put Linux Kernel source directory and external modules directory under the same directory. Here we call it `workspace` directory.
```shell
workspace
├── external_modules
└── linux
```
3. Open the `workspace` directory with VS code
```shell
$ code /your/path/to/workspace
```
4. Create `.vscode/launch.json` with the following content
Add `lx-symbols` command to `debugger_args`. `lx-symbols` command will recursively scan modules (.ko) under `workspace` directory.
```json
{
"version": "0.2.0",
"configurations": [
{
"type": "gdb",
"request": "attach",
"name": "Attach to gdbserver",
"gdbpath":"/your/path/to/gdb-multiarch",
"executable": "/your/path/to/vmlinux",
"debugger_args" : ["-ex", "lx-symbols"],
"target": "localhost:1234",
"remote": true,
"cwd": "${workspaceRoot}",
"valuesFormatting": "parseText"
}
]
}
```
5. Set a breakpoint for the `hellowolrd_init` function at file `workspace/external_modules/example_module/example_module.c` line 6, and then run debugger.
6. Load `example_module` in QEMU
```shell
# modprobe example_module
[ 46.537410] example_module: loading out-of-tree module taints kernel.
```
7. Wait for breakpoints to be triggered
8. Start debugging!
Sign in with Wallet
Connect another wallet