# Run Clang/LLVM Compiled Arm64 Linux Kernel on QEMU Virt with BusyBox Initramfs
[toc]
## Environment
### Host
* OS: Windows 11 Home Edition v10.0.22631
* WSL (Windows Subsystem for Linux)
* OS: Ubuntu 24.04.1 LTS
* Kernel: 5.15.153.1-microsoft-standard-WSL2
* Platform
* CPU: Intel i7-12700
* RAM: 32 GB
### Target
* OS: Linux
* Kernel: v6.6
* Platform: [QEMU Arm64 Virt](https://www.qemu.org/docs/master/system/arm/virt.html)
## BusyBox Initramfs
### Download toolchain and source files
1. Install `gcc` toolchain and check version
```shell
$ sudo apt-get install gcc
$ gcc --version
gcc (Ubuntu 13.2.0-23ubuntu4) 13.2.0
$ sudo apt-get install gcc-aarch64-linux-gnu
$ aarch64-linux-gnu-gcc --version
aarch64-linux-gnu-gcc (Ubuntu 13.2.0-23ubuntu4) 13.2.0
```
2. Download BusyBox v1.36.1
```shell!
$ wget https://busybox.net/downloads/busybox-1.36.1.tar.bz2
```
3. Unzip
```shell
$ tar -xvf busybox-1.36.1.tar.bz2
```
### Configure
1. Create `.config`
```shell
$ make ARCH=aarch64 CROSS_COMPILE=aarch64-linux-gnu- defconfig
```
2. Set more options in `.config` according to your needs
```shell
$ make ARCH=aarch64 CROSS_COMPILE=aarch64-linux-gnu- menuconfig
```
* (Must) Building BusyBox as a static binary (no shared libs)
```shell
CONFIG_STATIC=y
```
* (Optional) Workaround for this [bug](http://lists.busybox.net/pipermail/busybox-cvs/2024-January/041752.html)
```shell
# CONFIG_TC is not defined
```
### Build and Install
1. Build
```shell
$ make ARCH=aarch64 CROSS_COMPILE=aarch64-linux-gnu- \
all -j$(nproc)
```
2. Install
```shell
$ make ARCH=aarch64 CROSS_COMPILE=aarch64-linux-gnu- \
install
```
3. Check `busybox` is Arm64
```shell
$ file _install/bin/busybox
_install/bin/busybox: ELF 64-bit LSB executable, ARM aarch64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=927f32abddd51aa1dec4b0d58c5e8c186cff9935, for GNU/Linux 3.7.0, stripped
```
### Create initramfs image
1. Create `_install/etc/init.d/rcS` with the following content
```shell!=
#!/bin/sh
echo "Mounting /proc"
mkdir -p /proc
mount -t proc proc /proc
echo "Mounting /sys"
mkdir -p /sys
mount -t sysfs sysfs /sys
echo "Mounting /dev"
mkdir -p /dev
mount -t tmpfs tmpfs /dev
echo "Mounting /dev/pts"
mkdir -p /dev/pts
mount -t devpts devpts /dev/pts
echo "Mounting /tmp"
mkdir -p /tmp
mount -t tmpfs tmpfs /tmp
# Follow busybox/doc/mdev.txt to set up mdev
echo /sbin/mdev > /proc/sys/kernel/hotplug
mdev -s
echo -e \"Welcome to Arm64 Linux, boot took $(cut -d' ' -f1 /proc/uptime) seconds\"
```
2. Set `_install/etc/init.d/rcS` to executable
```shell
$ chmod +x _install/etc/init.d/rcS
```
3. Create `_install/etc/profile` with the following content
```shell
PATH=/bin:/sbin:/usr/bin:/usr/sbin
```
4. Create initramfs image with cpio and gzip
* (Recommended) For initramfs, replace `/linuxrc` with `/init`
```shell
$ cd _install
$ rm -f linuxrc
$ ln -s bin/busybox init
$ find . | cpio --quiet -H newc -o | gzip -9 -n > ../initramfs.cpio.gz
$ cd ..
```
* (Legacy) For [initrd](https://docs.kernel.org/admin-guide/initrd.html), keep `/linuxrc`
```shell
$ cd _install
$ find . | cpio --quiet -H newc -o | gzip -9 -n > ../initramfs.cpio.gz
$ cd ..
```
5. Check `initramfs.cpio.gz`
```shell
$ file initramfs.cpio.gz
initramfs.cpio.gz: gzip compressed data, max compression, from Unix, original size modulo 2^32 2231808
```
## Linux Kernel
### Download toolchain and source files
1. Install `clang` and `llvm` toolchain and check version
```shell
$ sudo apt-get install llvm
$ llvm-config --version
18.1.8
$ sudo apt-get install clang
$ clang --version
Ubuntu clang version 18.1.8 (++20240731025043+3b5b5c1ec4a3-1~exp1~20240731145144.92)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
```
2. Download Linux Kernel source code and checkout to v6.6
```shell
$ git clone https://github.com/torvalds/linux.git
$ git checkout v6.6
```
### Configure
1. Create `.config`
```shell
$ make ARCH=arm64 LLVM=1 LLVM_IAS=1 defconfig
```
2. Set more options in `.config` according to your needs
```shell
$ make ARCH=arm64 LLVM=1 LLVM_IAS=1 menuconfig
```
* (Recommended) For BusyBox `mdev`
```shell
CONFIG_UEVENT_HELPER=y
```
* (Recommended) For GDB and KGDB
```shell
CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_INFO_REDUCED is not set
CONFIG_FRAME_POINTER=y
CONFIG_GDB_SCRIPTS=y
CONFIG_KGDB=y
CONFIG_KGDB_SERIAL_CONSOLE=y
# CONFIG_RANDOMIZE_BASE is not set
```
* (Optional) If software breakpoints don't work well
```shell
# CONFIG_STRICT_KERNEL_RWX is not set
# CONFIG_STRICT_MODULE_RWX is not set
```
* (Optional) For [QEMU's 9pfs](https://wiki.qemu.org/Documentation/9psetup)
```shell
CONFIG_9P_FS=y
CONFIG_9P_FS_POSIX_ACL=y
CONFIG_NET_9P=y
CONFIG_NET_9P_VIRTIO=y
CONFIG_NET_9P_DEBUG=y
CONFIG_PCI=y
CONFIG_PCI_HOST_GENERIC=y
CONFIG_VIRTIO_PCI=y
```
### Build
1. Build
```shell
$ make ARCH=arm64 LLVM=1 LLVM_IAS=1 \
all -j$(nproc)
```
2. Check `vmlinux` and `arch/arm64/boot/Image` is Arm64
```shell
$ file vmlinux
vmlinux: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), statically linked, BuildID[sha1]=32e5529eeed74008343a46aa5a6fd5834b320b05, with debug_info, not stripped
$ file arch/arm64/boot/Image
arch/arm64/boot/Image: Linux kernel ARM64 boot executable Image, little-endian, 4K pages
```
## External Modules (Out-of-Tree Modules)
### Create source files
1. Create a directory out of Linux Kernel source directory
```shell
mkdir /your/path/to/externel_modules/example_module
```
2. Create source files with the following content
* `/your/path/to/externel_modules/example_module/example_module.c`
```c=
#include <linux/init.h>
#include <linux/module.h>
#include <linux/kernel.h>
static int __init hellowolrd_init(void) {
pr_info("Hello world!\n");
return 0;
}
static void __exit hellowolrd_exit(void) {
pr_info("End of the world\n");
}
module_init(hellowolrd_init);
module_exit(hellowolrd_exit);
MODULE_AUTHOR("visitor <visitor@gmail.com>");
MODULE_LICENSE("GPL");
```
* `/your/path/to/externel_modules/example_module/Kbuild`
```shell
obj-m += example_module.o
```
### Build and Install
1. Build with `M` option under Linux Kernel source directory
```shell
$ make -C /your/path/to/linux \
ARCH=arm64 LLVM=1 LLVM_IAS=1 \
M=/your/path/to/externel_modules/example_module \
modules -j$(nproc)
```
2. Install with `M` and `INSTALL_MOD_PATH` options under Linux Kernel source directory
```shell
$ make -C /your/path/to/linux \
ARCH=arm64 LLVM=1 LLVM_IAS=1 \
M=/your/path/to/externel_modules/example_module \
INSTALL_MOD_PATH=/your/path/to/busybox-1.36.1/_install \
modules_install
```
3. Check `example_module.ko` is installed in initramfs directory
```shell
$ cd /your/path/to/busybox-1.36.1/_install
$ file lib/modules/6.6.0/updates/example_module.ko
lib/modules/6.6.0/updates/example_module.ko: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), BuildID[sha1]=9cef715102fc011a004d38d03514929a82a51be2, with debug_info, not stripped
```
### Update initramfs image
1. Update initramfs image
```shell
$ cd /your/path/to/busybox-1.36.1/_install
$ find . | cpio --quiet -H newc -o | gzip -9 -n > ../initramfs.cpio.gz
```
## QEMU
### Install
```shell
$ sudo apt-get install qemu-system
$ qemu-system-aarch64 -version
QEMU emulator version 8.2.2 (Debian 1:8.2.2+ds-0ubuntu1.2)
```
### Run
* (Recommended) Boot with initramfs, which runs `/init` during boot
```shell
$ qemu-system-aarch64 \
-machine virt \
-m size=1024M \
-cpu cortex-a57 \
-smp 2 \
-nographic \
-initrd /your/path/to/busybox-1.36.1/initramfs.cpio.gz \
-kernel /your/path/to/linux/arch/arm64/boot/Image \
--append "console=ttyAMA0"
```
* (Legacy) Boot with initrd, which runs `/linuxrc` during boot and needs Kernel bootargs `rdinit` and `root`
```shell
$ qemu-system-aarch64 \
-machine virt \
-m size=1024M \
-cpu cortex-a57 \
-smp 2 \
-nographic \
-initrd /your/path/to/busybox-1.36.1/initramfs.cpio.gz \
-kernel /your/path/to/linux/arch/arm64/boot/Image \
--append "console=ttyAMA0 rdinit=/linuxrc root=/dev/ram0"
```
### Boot to Linux and load/unload a externel module
```shell
# modprobe example_module
# modprobe -r example_module
```
### Device Tree
QEMU uses default device tree blob
1. Install toolchain
```shell
$ sudo apt-get install device-tree-compiler
```
2. Dump device tree blob `.dtb`
Append `dumpdtb=qemu_virt_arm64.dtb` in `-machine` option
```shell
$ qemu-system-aarch64 \
-machine virt,dumpdtb=qemu_virt_arm64.dtb \
-m size=1024M \
-cpu cortex-a57 \
-smp 2 \
-nographic \
-initrd /your/path/to/busybox-1.36.1/initramfs.cpio.gz \
-kernel /your/path/to/linux/arch/arm64/boot/Image \
--append "console=ttyAMA0"
```
3. Compile device tree source `.dts`
```shell
$ dtc -I dtb -O dts -o qemu_virt_arm64.dts qemu_virt_arm64.dtb
```
4. Compile device tree blob `.dtb`
```shell
$ dtc -@ -I dts -O dtb -o qemu_virt_arm64.dtb qemu_virt_arm64.dts
```
5. Run QEMU with `-dtb` option
```shell
$ qemu-system-aarch64 \
-machine virt \
-m size=1024M \
-cpu cortex-a57 \
-smp 2 \
-nographic \
-dtb /your/path/to/qemu_virt_arm64.dtb \
-initrd /your/path/to/busybox-1.36.1/initramfs.cpio.gz \
-kernel /your/path/to/linux/arch/arm64/boot/Image \
--append "console=ttyAMA0"
```
## Misc.
### Clangd
A powerful language server for Linux Kernel developers, please visit the [clangd official website](https://clangd.llvm.org/) for more details.
1. Install `clangd`
```shell
$ sudo apt-get install clangd-12
```
2. Generate `compile_commands.json` for Linux Kernel
```shell
$ make -C /your/path/to/linux \
ARCH=arm64 LLVM=1 LLVM_IAS=1 \
compile_commands.json
```
3. Generate `compile_commands.json` with `M` option for externel modules
```shell
$ make -C /your/path/to/linux \
ARCH=arm64 LLVM=1 LLVM_IAS=1 \
M=/your/path/to/externel_modules/example_module \
compile_commands.json
```
Sign in with Wallet
Connect another wallet