---
# System prepended metadata

title: Run Clang/LLVM Compiled Arm64 Linux Kernel on QEMU Virt with BusyBox Initramfs
tags: [QEMU, Linux Kernel]

---

# Run Clang/LLVM Compiled Arm64 Linux Kernel on QEMU Virt with BusyBox Initramfs  
[toc]  
## Environment  
### Host  
* OS: Windows 11 Home Edition v10.0.22631  
* WSL (Windows Subsystem for Linux)  
    * OS: Ubuntu 24.04.1 LTS  
    * Kernel: 5.15.153.1-microsoft-standard-WSL2  
* Platform  
    * CPU: Intel i7-12700  
    * RAM: 32 GB  
### Target  
* OS: Linux  
* Kernel: v6.6  
* Platform: [QEMU Arm64 Virt](https://www.qemu.org/docs/master/system/arm/virt.html)  
## BusyBox Initramfs  
### Download toolchain and source files  
1. Install `gcc` toolchain and check version  
    ```shell
    $ sudo apt-get install gcc
    $ gcc --version
    gcc (Ubuntu 13.2.0-23ubuntu4) 13.2.0
    
    $ sudo apt-get install gcc-aarch64-linux-gnu
    $ aarch64-linux-gnu-gcc --version
    aarch64-linux-gnu-gcc (Ubuntu 13.2.0-23ubuntu4) 13.2.0
    ```
2. Download BusyBox v1.36.1  
    ```shell!
    $ wget https://busybox.net/downloads/busybox-1.36.1.tar.bz2
    ```
3. Unzip  
    ```shell
    $ tar -xvf busybox-1.36.1.tar.bz2
    ```
### Configure  
1. Create `.config`  
    ```shell
    $ make ARCH=aarch64 CROSS_COMPILE=aarch64-linux-gnu- defconfig
    ```
2. Set more options in `.config` according to your needs  
    ```shell
    $ make ARCH=aarch64 CROSS_COMPILE=aarch64-linux-gnu- menuconfig
    ```
    * (Must) Building BusyBox as a static binary (no shared libs)   
    ```shell
    CONFIG_STATIC=y
    ```
    * (Optional) Workaround for this [bug](http://lists.busybox.net/pipermail/busybox-cvs/2024-January/041752.html)  
    ```shell
    # CONFIG_TC is not defined
    ```
### Build and Install  
1. Build  
    ```shell
    $ make ARCH=aarch64 CROSS_COMPILE=aarch64-linux-gnu- \
           all -j$(nproc)
    ```
2. Install  
    ```shell
    $ make ARCH=aarch64 CROSS_COMPILE=aarch64-linux-gnu- \
           install
    ```
3. Check `busybox` is Arm64  
    ```shell
    $ file _install/bin/busybox
    _install/bin/busybox: ELF 64-bit LSB executable, ARM aarch64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=927f32abddd51aa1dec4b0d58c5e8c186cff9935, for GNU/Linux 3.7.0, stripped
    ```
### Create initramfs image  
1. Create `_install/etc/init.d/rcS` with the following content  
    ```shell!=
    #!/bin/sh
    
    echo "Mounting /proc"
    mkdir -p /proc
    mount -t proc proc /proc
    
    echo "Mounting /sys"
    mkdir -p /sys
    mount -t sysfs sysfs /sys
    
    echo "Mounting /dev"
    mkdir -p /dev
    mount -t tmpfs tmpfs /dev
    
    echo "Mounting /dev/pts"
    mkdir -p /dev/pts
    mount -t devpts devpts /dev/pts
    
    echo "Mounting /tmp"
    mkdir -p /tmp
    mount -t tmpfs tmpfs /tmp
    
    # Follow busybox/doc/mdev.txt to set up mdev
    echo /sbin/mdev > /proc/sys/kernel/hotplug
    mdev -s
    
    echo -e \"Welcome to Arm64 Linux, boot took $(cut -d' ' -f1 /proc/uptime) seconds\"
    
    ```
2. Set `_install/etc/init.d/rcS` to executable  
    ```shell
    $ chmod +x _install/etc/init.d/rcS
    ```
3. Create `_install/etc/profile` with the following content  
    ```shell
    PATH=/bin:/sbin:/usr/bin:/usr/sbin
    ```
4. Create initramfs image with cpio and gzip  
    * (Recommended) For initramfs, replace `/linuxrc` with `/init`  
    ```shell
    $ cd _install
    $ rm -f linuxrc
    $ ln -s bin/busybox init
    $ find . | cpio --quiet -H newc -o | gzip -9 -n > ../initramfs.cpio.gz
    $ cd ..
    ```
    * (Legacy) For [initrd](https://docs.kernel.org/admin-guide/initrd.html), keep `/linuxrc`  
    ```shell
    $ cd _install
    $ find . | cpio --quiet -H newc -o | gzip -9 -n > ../initramfs.cpio.gz
    $ cd ..
    ```
5. Check `initramfs.cpio.gz`  
    ```shell
    $ file initramfs.cpio.gz
    initramfs.cpio.gz: gzip compressed data, max compression, from Unix, original size modulo 2^32 2231808
    ```
## Linux Kernel  
### Download toolchain and source files  
1. Install `clang` and `llvm` toolchain and check version  
    ```shell
    $ sudo apt-get install llvm
    $ llvm-config  --version
    18.1.8
    
    $ sudo apt-get install clang
    $ clang --version
    Ubuntu clang version 18.1.8 (++20240731025043+3b5b5c1ec4a3-1~exp1~20240731145144.92)
    Target: x86_64-pc-linux-gnu
    Thread model: posix
    InstalledDir: /usr/bin
    ```
2. Download Linux Kernel source code and checkout to v6.6  
    ```shell
    $ git clone https://github.com/torvalds/linux.git
    $ git checkout v6.6
    ```
### Configure  
1. Create `.config`  
    ```shell
    $ make ARCH=arm64 LLVM=1 LLVM_IAS=1 defconfig
    ```
2. Set more options in `.config` according to your needs  
    ```shell
    $ make ARCH=arm64 LLVM=1 LLVM_IAS=1 menuconfig
    ```
    * (Recommended) For BusyBox `mdev`  
    ```shell
    CONFIG_UEVENT_HELPER=y
    ```
    * (Recommended) For GDB and KGDB  
    ```shell
    CONFIG_DEBUG_INFO=y
    # CONFIG_DEBUG_INFO_REDUCED is not set
    CONFIG_FRAME_POINTER=y
    CONFIG_GDB_SCRIPTS=y
    CONFIG_KGDB=y
    CONFIG_KGDB_SERIAL_CONSOLE=y
    # CONFIG_RANDOMIZE_BASE is not set
    ```
    * (Optional) If software breakpoints don't work well  
    ```shell
    # CONFIG_STRICT_KERNEL_RWX is not set
    # CONFIG_STRICT_MODULE_RWX is not set
    ```
    * (Optional) For [QEMU's 9pfs](https://wiki.qemu.org/Documentation/9psetup)  
    ```shell
    CONFIG_9P_FS=y
    CONFIG_9P_FS_POSIX_ACL=y
    CONFIG_NET_9P=y
    CONFIG_NET_9P_VIRTIO=y
    CONFIG_NET_9P_DEBUG=y
    CONFIG_PCI=y
    CONFIG_PCI_HOST_GENERIC=y
    CONFIG_VIRTIO_PCI=y
    ```
### Build  
1. Build  
    ```shell
    $ make ARCH=arm64 LLVM=1 LLVM_IAS=1 \
           all -j$(nproc)
    ```
2. Check `vmlinux` and `arch/arm64/boot/Image` is Arm64  
    ```shell
    $ file vmlinux
    vmlinux: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), statically linked, BuildID[sha1]=32e5529eeed74008343a46aa5a6fd5834b320b05, with debug_info, not stripped
    
    $ file arch/arm64/boot/Image
    arch/arm64/boot/Image: Linux kernel ARM64 boot executable Image, little-endian, 4K pages
    ```
## External Modules (Out-of-Tree Modules)  
### Create source files  
1. Create a directory out of Linux Kernel source directory  
    ```shell
    mkdir /your/path/to/externel_modules/example_module
    ```
2. Create source files with the following content  
    * `/your/path/to/externel_modules/example_module/example_module.c`  
    ```c=
    #include <linux/init.h>
    #include <linux/module.h>
    #include <linux/kernel.h>
    
    static int __init hellowolrd_init(void) {
        pr_info("Hello world!\n");
        return 0;
    }
    
    static void __exit hellowolrd_exit(void) {
        pr_info("End of the world\n");
    }
    
    module_init(hellowolrd_init);
    module_exit(hellowolrd_exit);
    MODULE_AUTHOR("visitor <visitor@gmail.com>");
    MODULE_LICENSE("GPL");
    ```
    * `/your/path/to/externel_modules/example_module/Kbuild`  
    ```shell
    obj-m += example_module.o
    ```
### Build and Install  
1. Build with `M` option under Linux Kernel source directory  
    ```shell
    $ make -C /your/path/to/linux \
           ARCH=arm64 LLVM=1 LLVM_IAS=1 \
           M=/your/path/to/externel_modules/example_module \
           modules -j$(nproc)
    ```
2. Install with `M` and `INSTALL_MOD_PATH` options under Linux Kernel source directory  
    ```shell
    $ make -C /your/path/to/linux \
           ARCH=arm64 LLVM=1 LLVM_IAS=1 \
           M=/your/path/to/externel_modules/example_module \
           INSTALL_MOD_PATH=/your/path/to/busybox-1.36.1/_install \
           modules_install
    ```
3. Check `example_module.ko` is installed in initramfs directory  
    ```shell
    $ cd /your/path/to/busybox-1.36.1/_install
    $ file lib/modules/6.6.0/updates/example_module.ko
    lib/modules/6.6.0/updates/example_module.ko: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), BuildID[sha1]=9cef715102fc011a004d38d03514929a82a51be2, with debug_info, not stripped
    ```
### Update initramfs image  
1. Update initramfs image  
    ```shell
    $ cd /your/path/to/busybox-1.36.1/_install
    $ find . | cpio --quiet -H newc -o | gzip -9 -n > ../initramfs.cpio.gz
    ```
## QEMU  
### Install  
```shell
$ sudo apt-get install qemu-system
$ qemu-system-aarch64 -version
QEMU emulator version 8.2.2 (Debian 1:8.2.2+ds-0ubuntu1.2)
```
### Run  
* (Recommended) Boot with initramfs, which runs `/init` during boot  
    ```shell
    $ qemu-system-aarch64 \
          -machine virt \
          -m size=1024M \
          -cpu cortex-a57 \
          -smp 2 \
          -nographic \
          -initrd /your/path/to/busybox-1.36.1/initramfs.cpio.gz \
          -kernel /your/path/to/linux/arch/arm64/boot/Image \
          --append "console=ttyAMA0"
    ```
* (Legacy) Boot with initrd, which runs `/linuxrc` during boot and needs Kernel bootargs `rdinit` and `root`  
    ```shell
    $ qemu-system-aarch64 \
          -machine virt \
          -m size=1024M \
          -cpu cortex-a57 \
          -smp 2 \
          -nographic \
          -initrd /your/path/to/busybox-1.36.1/initramfs.cpio.gz \
          -kernel /your/path/to/linux/arch/arm64/boot/Image \
          --append "console=ttyAMA0 rdinit=/linuxrc root=/dev/ram0"
    ```
### Boot to Linux and load/unload a externel module  
```shell
# modprobe example_module
# modprobe -r example_module
```
![image](https://hackmd.io/_uploads/S10xHVyokl.png)  
### Device Tree  
QEMU uses default device tree blob  
1. Install toolchain  
    ```shell
    $ sudo apt-get install device-tree-compiler
    ```
2. Dump device tree blob `.dtb`  
    Append `dumpdtb=qemu_virt_arm64.dtb` in `-machine` option
    ```shell
    $ qemu-system-aarch64 \
          -machine virt,dumpdtb=qemu_virt_arm64.dtb \
          -m size=1024M \
          -cpu cortex-a57 \
          -smp 2 \
          -nographic \
          -initrd /your/path/to/busybox-1.36.1/initramfs.cpio.gz \
          -kernel /your/path/to/linux/arch/arm64/boot/Image \
          --append "console=ttyAMA0"
    ```
3. Compile device tree source `.dts`  
    ```shell
   $ dtc -I dtb -O dts -o qemu_virt_arm64.dts qemu_virt_arm64.dtb
    ```
4. Compile device tree blob `.dtb`  
    ```shell
    $ dtc -@ -I dts -O dtb -o qemu_virt_arm64.dtb qemu_virt_arm64.dts
    ```
5. Run QEMU with `-dtb` option  
    ```shell
    $ qemu-system-aarch64 \
          -machine virt \
          -m size=1024M \
          -cpu cortex-a57 \
          -smp 2 \
          -nographic \
          -dtb    /your/path/to/qemu_virt_arm64.dtb \
          -initrd /your/path/to/busybox-1.36.1/initramfs.cpio.gz \
          -kernel /your/path/to/linux/arch/arm64/boot/Image \
          --append "console=ttyAMA0"
    ```
## Misc.  
### Clangd  
A powerful language server for Linux Kernel developers, please visit the [clangd official website](https://clangd.llvm.org/) for more details.  
1. Install `clangd`  
   ```shell
   $ sudo apt-get install clangd-12
   ```
2. Generate `compile_commands.json` for Linux Kernel  
    ```shell
    $ make -C /your/path/to/linux \
           ARCH=arm64 LLVM=1 LLVM_IAS=1 \
           compile_commands.json
    ```
3. Generate `compile_commands.json` with `M` option for externel modules  
   ```shell
    $ make -C /your/path/to/linux \
           ARCH=arm64 LLVM=1 LLVM_IAS=1 \
           M=/your/path/to/externel_modules/example_module \
           compile_commands.json
    ```
