# Run [kbox](https://github.com/sysprog21/kbox/) on riscv64 contributed by < [`rota1001`](https://github.com/rota1001) > ## Run it on AARCH64 I run it on ARCH64 architecture first and then change to riscv64 ### Compile Linux Kernel ```shell $ wget https://www.kernel.org/pub/linux/kernel/v6.x/linux-6.19.9.tar.xz $ tar Jxvf linux-6.19.9.tar.xz $ cd linux-6.19.9 $ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- defconfig $ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- -j`nproc` ``` ### Build rootfs ```shell $ wget https://dl-cdn.alpinelinux.org/alpine/v3.21/releases/aarch64/alpine-minirootfs-3.21.0-aarch64.tar.gz $ dd if=/dev/zero of=aarch64_alpine_host.ext4 bs=1M count=256 $ mkfs.ext4 aarch64_alpine_host.ext4 $ mkdir rootfs $ sudo mount aarch64_alpine_host.ext4 rootfs $ sudo tar -xvf alpine-minirootfs-3.21.0-aarch64.tar.gz -C rootfs $ sudo umount rootfs ``` ### Build kbox Build it under the `kbox/` directory: ```shell $ make ARCH=aarch64 CC=aarch64-linux-gnu-gcc BUILD=release $ make ARCH=aarch64 CC=aarch64-linux-gnu-gcc rootfs ``` Put kbox and alpine.ext4 into rootfs: ```shell $ sudo mount aarch64_alpine_host.ext4 rootfs $ sudo cp kbox rootfs $ sudo cp alpine.ext4 rootfs ``` Since the dynamic link library in alpine is musl rather than glibc but the pre-compiled LKL uses glibc, the glibc dynamic link library has to be copied into the rootfs. ```shell $ sudo cp -a /usr/aarch64-linux-gnu/lib/* rootfs/lib $ sudo umount rootfs ``` ### Run kbox Run the Linux kernel in QEMU： ```shell $ qemu-system-aarch64 \ -machine virt \ -cpu cortex-a53 \ -m 2G \ -nographic \ -kernel linux-6.19.9/arch/arm64/boot/Image \ -append "root=/dev/vda rw console=ttyAMA0 init=/bin/sh" \ -drive file=aarch64_alpine_host.ext4,format=raw,id=hd0,if=none \ -device virtio-blk-device,drive=hd0 ``` After successfully boot, mount the procfs, which is needed by kbox： ```shell $ mount -t proc none /proc ``` Run the kbox： ```shell ~ # ./kbox image -S alpine.ext4 -- /bin/sh mount(devtmpfs on /dev): No such device warning: devtmpfs unavailable (CONFIG_DEVTMPFS?) kbox: probing host features... kbox: all feature probes passed. /bin/sh: can't access tty; job control turned off ~ # uname -a Linux kbox 6.8.0-kbox #1 SMP aarch64 Linux ``` And it works well. <!-- ```shell $ wget https://dl-cdn.alpinelinux.org/alpine/v3.21/releases/aarch64/alpine-virt-3.21.0-aarch64.iso ``` 創建一個硬碟，裝系統： ```shell $ qemu-img create -f qcow2 disk.qcow2 10G $ qemu-system-aarch64 \ -machine virt \ -cpu cortex-a53 \ -m 1G \ -nographic \ -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd \ -cdrom alpine-virt-3.21.0-aarch64.iso \ -drive file=disk.qcow2,format=qcow2,if=virtio ``` 把系統裝到 disk.qcow2 上。 接下來要利用 VirtFS 來達成 qemu 機器與宿主機間的目錄共享： ```shell $ mkdir share $ echo "meow" > meow.txt $ qemu-system-aarch64 \ -machine virt \ -cpu cortex-a53 \ -m 1G \ -nographic \ -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd \ -drive file=alpine-disk.qcow2,format=qcow2,if=virtio \ -fsdev local,security_model=passthrough,id=fsdev0,path=`pwd`/share \ -device virtio-9p-pci,id=fs0,fsdev=fsdev0,mount_tag=share ... rota1001:~# mount -t 9p -o trans=virtio,version=9p2000.L share /share rota1001:~# ls /share meow.txt rota1001:~# cat /share/meow.txt meow ``` 手動 mount 成功後，修改 fstab 來達成開機自動掛載： ```shell $ echo "share /share 9p trans=virtio,version=9p2000.L,rw 0 0" >> /etc/ fstab ``` 以上的啟動腳本我寫在 `run.sh` 這個 shellscript 中，以下會直接使用它。 編譯 kbox： ```shell $ cd kbox/ $ scripts/fetch-lkl.sh aarch64 $ make LKL_DIR=`pwd`/lkl-aarch64 CC=aarch64-linux-gnu-gcc BUILD=release $ make rootfs ALPINE_ARCH=aarch64 ``` 把 kbox 與 rootfs 放進 share 中： ```shell $ cp kbox/kbox kbox/alpine.ext4 share ``` 在 alpine 中預設是使用 musl 而非 glibc，但 LKL 與 kbox 使用的都是 glibc，所以要把動態連結庫搬到裡面去，首先把他們放到 share 中： ```shell $ cp -r /usr/aarch64-linux-gnu/lib share ``` 啟動後搬運動態連結庫： ```shell rota1001:~# cp -a /share/lib/* /lib ``` 搬運完後就能啟動了： ```shell rota1001:~# /share/kbox image -S /share/alpine.ext4 -- /bin/sh -i kbox: probing host features... kbox: all feature probes passed. /bin/sh: can't access tty; job control turned off / # uname -a Linux kbox 6.8.0-kbox #1 SMP aarch64 Linux# ``` --> ## Run it on riscv64 ### Compile LKL In the mainline LKL, it takes big amount of time for link time relaxation while running on the riscv64 architecture, so I used my own patch version (,btw, my [PR](https://github.com/lkl/linux/pull/621) is on the way) ```shell $ sudo dpkg --add-architecture riscv64 $ sudo apt update $ sudo apt install -y libfuse3-dev:riscv64 libarchive-dev:riscv64 $ wget https://github.com/rota1001/lkl/archive/refs/heads/rv64.zip $ unzip rv64.zip $ mv lkl-rv64 linux && cd linux $ make ARCH=lkl menuconfig ``` Do the following settings in menuconfig: - Change the Output format to `elf64-littleriscv` - Set the Device Drivers -> Generic Driver Options -> Maintain a devtmpfs filesystem to mount at /dev - Set the Device Drivers -> Generic Driver Options -> Automount devtmpfs at /dev, after the kernel mounted the rootfs Then compile: ```shell $ make CROSS_COMPILE=riscv64-linux-gnu- -C tools/lkl -j`nproc` ``` ### Compile the Linux Kernel ```shell $ wget https://www.kernel.org/pub/linux/kernel/v6.x/linux-6.19.9.tar.xz $ tar Jxvf linux-6.19.9.tar.xz $ cd linux-6.19.9 $ make ARCH=riscv CROSS_COMPILE=riscv64-linux-gnu- defconfig $ make ARCH=riscv CROSS_COMPILE=riscv64-linux-gnu- -j`nproc` ``` ### Build rootfs ```shell $ wget https://dl-cdn.alpinelinux.org/alpine/v3.21/releases/riscv64/alpine-minirootfs-3.21.0-riscv64.tar.gz $ dd if=/dev/zero of=riscv64_alpine_host.ext4 bs=1M count=256 $ mkfs.ext4 riscv64_alpine_host.ext4 $ mkdir rootfs $ sudo mount riscv64_alpine_host.ext4 rootfs $ sudo tar -xvf alpine-minirootfs-3.21.0-riscv64.tar.gz -C rootfs $ sudo umount rootfs ``` ### Build kbox kbox needs some patch here, so I use my own patch: ```shell $ git clone https://github.com/rota1001/kbox.git $ cd kbox $ git checkout rv64 ``` Now, we are in `kbox/` directory and the LKL source code is in `kbox/linux`. Let's start building the kbox: ``` $ make LKL_DIR=linux/tools/lkl ARCH=riscv64 CC=riscv64-linux-gnu-gcc BUILD=release $ make ARCH=riscv64 CC=riscv64-linux-gnu-gcc rootfs ``` Put both kbox and apline.ext4 into the rootfs: ```shell $ sudo mount riscv64_alpine_host.ext4 rootfs $ sudo cp kbox rootfs $ sudo cp alpine.ext4 rootfs ``` Copy the dynamic linked library into it: ```shell $ sudo cp -a /usr/riscv64-linux-gnu/lib/* rootfs/lib $ sudo umount rootfs ``` ### Run kbox Run the Linux Kernel in QEMU: ```shell $ qemu-system-riscv64 \ -machine virt \ -cpu rv64 \ -m 2G \ -nographic \ -kernel linux-6.19.9/arch/riscv/boot/Image \ -append "root=/dev/vda rw console=ttyS0 init=/bin/sh" \ -drive file=riscv64_alpine_host.ext4,format=raw,id=hd0 \ -device virtio-blk-device,drive=hd0 ``` Mount the procfs： ```shell $ mount -t proc none /proc ``` Run the kbox: ```shell ~ # ./kbox image -S alpine.ext4 -- /bin/sh kbox: probing host features... kbox: all feature probes passed. /bin/sh: can't access tty; job control turned off ~ # uname -a Linux kbox 6.8.0-kbox #1 SMP unknown Linux ``` And it works well.