QEMU for Beginner and Advanced

tags: `2022/09` `qemu` `qemu-user` `qemu-architecture` `qemu-system-architecture`

(2022/9/18) QEMU for beginner, starting from few simple examples so to reduce the hurdle to get familiar with qemu.
(latest update on 2022/10/1)

Table of Contents

QEMU for Beginner and Advanced
QEMU for Advanced
References

What is qemu?

From Linux Q&A-What is qemu?, we learn that there are 3 modes supported by qmue:

A user mode emulator : QEMU can launch Linux processes compiled for one CPU on another CPU, translating syscalls on the fly. This QEMU mode is faster than full system emulation, but is not a perfect abstraction.
A full system emulator : QEMU emulates a full system (virtual machine), including a processor and various peripherals such as disk, ethernet controller etc. That is normally how an operating system works.
A virtualization environment : used by KVM and XEN virtualization environments. (Will not go into details in this article)

QEMU user mode
It is much more easier to try with QEMU user mode. Users just need to be familiar with Linux shell command, and a basic programming skill, hello world level, in C language. We will start with 2 examples :

First one for QEMU to run x86 simulation. People might wonder why we need to use x86 system to simulate x86 CPU? The purpose is just to demonstrate the function of QEMU as a stepping stone to simulate another CPU of ARM. It is a very good entry to run QEMU. We just need to install gcc for x86 environment.
Second example is for QEMU to run ARM simulation. Similar to first example, but needs to install ARM cross compiler first to generate ARM executive binary program.

QEMU system mode
After then, the third example will be running QEMU system mode. To run QEMU system mode, it requires an image file containing a complete operating system to run, instead of a single Linux program to run on QEMU user mode. First we pick an old Linux version running on i386. Then pick the target of Raspberry Pi, which is a very popular SBC (Single Board Computer) based on ARM CPU. We will use the Raspberry Pi image, running Debian based Linux called Raspbian, on QEMU system mode simulation.

First QEMU system mode example is to simulate i386 to run old Linux version 0.11.
Second QEMU system mode example is to simulate ARM CPU to run Raspberry Pi image.

QEMU Installation

One (easy, but not recommended) way to install qemu is to use package installation. It will install both both user and system modes of qemu. However, it installed qemu version 2.5.0 on Ubuntu 16.04, which is relatively old version.

# Ubuntu environment
sudo apt install qemu

Recommend to use newer version, by following instruction from https://www.qemu.org/download/. By the time I download, the latest version was 7.1.0. It requires Python 3.8 than Ubuntu 16.04 default Python 3.5. So I chose version 5.0.0 arbitrarily which just works.

wget https://download.qemu.org/qemu-5.0.0.tar.xz
tar xvJf qemu-5.0.0.tar.xz
cd qemu-5.0.0
sudo apt update
sudo apt-get install libgtk-3-dev
./configure --enable-gtk
sudo make install

Check Linux Q&A: QEMU - nothing shows up after QEMU VNC server running for explanation of qemu ./configure option.

QEMU Example 1 : `qemu-x86_64` and `qemu-i386`

Running `hello world` on Linux native environmemt

With x86_64 system used for Linux, qemu is not really needed to run x86_64 programs like hello_world. However, we still can use qemu to simulate x86_64 program, like we do for native x86_64 program.

Let's start with hello world program running on Linux native x86_64 environment.

# install gcc to build hello work program
sudo apt update
sudo apt install build-essential

use vi or other editor program to create hello.c file content like below

#include <stdio.h>

int main() {
    printf("Hello Marconi...\n");
    return(0);
}

then compile and execute hello program on Linux

gcc hello.c -o hello
./hello
# output below
Hello Marconi...

# or compile to 32bit object code
gcc -m32 hello.c -o hello32
./hello32
# output below
Hello Marconi...

We can check the file attribute with Linux command file to see the attritube of hello is executable at x86_64 ELF 64-bit LSB mode, and hello32 is executable at Intel 80386 (nornally use i386 for short) ELF 32-bit LSB mode.

file hello
# output below
hello: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, 
interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32,
BuildID[sha1]=51f76d8ea67c7fd9506745a09f8698e552d05933, not stripped

file hello32
# output below
hello32: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked, 
interpreter /lib/ld-linux.so.2, for GNU/Linux 2.6.32,
BuildID[sha1]=50e617ac0122622e4d1ced8079f9a2d19868b148, not stripped

Running `hello world` on qemu

First, we install qemu user mode and run the hello program.

# Install qemu user mode environment
sudo apt install qemu-user

# Check what are installed
ls /user/bin/qemu
# the output will include many architectures qemu user mode support. 
# Only few are listed below
/usr/bin/qemu-aarch64
/usr/bin/qemu-arm
/usr/bin/qemu-i386
/usr/bin/qemu-x86_64
....

Keep in mind that the naming convention of qemu user mode command is qemu-arctecture where architecture can be arm (ARM 32bit version), aarch64 (ARM 64bit), i386 (x86 32bit), x86_64 (x86 64 bit), etc. In next section, we will introduction qemu system mode. The naming convention is qemu-system-architecture. You can see what the difference is.

Let's see how we can execute the hello programs on qemu.

# running hello program under qemu-x86_64 environment
qemu-x86_64 ./hello

# output below. We can ignore the warning at this moment
warning: TCG doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5]
Hello Marconi...

# or
# running 32bit version of hello program under qemu-i386 environment
qemu-i386 ./hello

# output below. 
Hello Marconi...

Till now, it is quite easy and straight forward to use qemu to simulate and run x86 programs.

QEMU Example 2 : `qemu-aarch64`

To run QEMU simulation for ARM CPU, we need, at least, a hello world program in ARM binary code to demostrate qemu-arm and qemu-aarch64 features. We are going to build the hello world binary by installing the cross compiler of gcc for ARM first.

Install ARM toolchain

We can build ARM binary code on x86 CPU with cross compiler. 'Cross' means to build a binary code for one CPU when it is running on another CPU. x86 is the one of the most popular and power CPU. We can use it to build the binary code for another CPU, like ARM, RISC-V, or other embedded types of CPU, which can save development and compilation time due to powerful x86 CPU, and many tools available.

According to wikipedia, a toolchain is a set of programming tools that is used to perform a complex software development task or to create a software product, which is typically another computer program or a set of related programs. In general, the tools forming a toolchain are executed consecutively so the output or resulting environment state of each tool becomes the input or starting environment for the next one, but the term is also used when referring to a set of related tools that are not necessarily executed consecutively.

A simple software development toolchain may consist of a compiler and linker (which transform the source code into an executable program), libraries (which provide interfaces to the operating system), and a debugger (which is used to test and debug created programs).

We use the following command to install ARM64 gcc tool chain, using the same hello.c as in Running hello world on Linux native environmemt

sudo apt update
# install gcc for ARM64 (aarch64)
sudo apt install gcc-aarch64-linux-gnu

# compile hello.c with aarch64 gcc
# the same hello.c as in 
aarch64-linux-gnu-gcc hello.c -static -o hello-arm64

# check file attribute of hello-arm64
file hello-arm64
# output
hello-arm64: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), 
statically linked, for GNU/Linux 3.7.0,
BuildID[sha1]=5cd2c9e6081bda47b83b196b41ff7c1a1113a850, not stripped

Now we have a executable ARM64 binary of hello-arm64, which can be executed on QEMU. We can find the file attribute of hello-arm64 is ELF 64-bit LSB executable based on ARM aarch64.

# check if qemu-aarch64 is installed?
ls /usr/bin/qemu-aarch64
# output
/usr/bin/qemu-aarch64

# execute hello-arm64 under qemu
qemu-aarch64 ./hello-arm64
# output 
Hello Marconi...

Now we succeed in running qemu-architecture to simulate ARM CPU in user mode. Next is to run qemu-system-architecture in system mode.

QEMU Example 3 : `qemu-system-i386`

It requires an OS image to run in QEMU system mode. We use old Linux 0.11 version as it is relatively simple.

Check the following www.oldlinux.org web and copy the link of file 'Linux 0.11 on qemu-12.5.i386.zip'

http://www.oldlinux.org/
- Linux Ancient Resources – Early and old Linux docs, source code and bins. (Linux.old)
  - qemu-images/
    - 'Linux 0.11 on qemu-12.5.i386.zip'

The link is http://www.oldlinux.org/Linux.old/qemu-images/Linux 0.11 on qemu-12.5.i386.zip.

Now follow the commands to execute qemu-system-386.

# go to the working directory of your own
cd ~/myworks
wget http://www.oldlinux.org/Linux.old/qemu-images/Linux%200.11%20on%20qemu-12.5.i386.zip

# unzip the file will create a new directory `qemu-12.5.i386`
unzip http://www.oldlinux.org/Linux.old/qemu-images/Linux%200.11%20on%20qemu-12.5.i386.zip

cd qemu-12.5.i386

# use the following command, reference from file `linux.bat`
qemu-system-i386 -hda linux-0.11-devel-060625.qcow2 -no-reboot -m 16M

# qemu will create a new ternimal screen, and start booting process.

Then enter key `1` to boot from the first device.

Now Linux 0.11 is running on qemu-system-i386. You may find many commands not found, like uname, lspci, lscpu, etc. Those available are vi, awk, fdisk, but there are i386 assembler as86, linker ld86 programs available, and gcc and hello.c as well.

QEMU Example 4 : `qemu-system-arm` running Raspberry Pi in command mode

Now come to more challenging one. I failed so many time trying to get Raspberry Pi image working. The easiest way is to create a script file of the following content, from this article.

#!/bin/sh

QEMU=$(command -v qemu-system-arm)
TMP_DIR=$(pwd)
RPI_KERNEL=kernel-qemu-4.19.50-buster
RPI_KERNEL_FILE=$TMP_DIR/$RPI_KERNEL
PTB=versatile-pb.dtb
PTB_FILE=$TMP_DIR/$PTB
IMAGE_BASE=2019-09-26-raspbian-buster-lite
IMAGE=$IMAGE_BASE.zip
IMAGE_FILE=$TMP_DIR/$IMAGE
RPI_FS=$TMP_DIR/$IMAGE_BASE.img

mkdir -p $TMP_DIR

wget https://github.com/dhruvvyas90/qemu-rpi-kernel/blob/master/${RPI_KERNEL}?raw=true \
        -O ${RPI_KERNEL_FILE}

wget https://github.com/dhruvvyas90/qemu-rpi-kernel/raw/master/$PTB \
        -O ${PTB_FILE}

wget http://downloads.raspberrypi.org/raspbian_lite/images/raspbian_lite-2019-09-30/$IMAGE \
        -O ${IMAGE_FILE}
unzip $IMAGE_FILE -d $TMP_DIR


$QEMU -kernel ${RPI_KERNEL_FILE} \
    -cpu arm1176 -m 256 -M versatilepb \
    -dtb ${PTB_FILE} -no-reboot \
    -serial stdio -append "root=/dev/sda2 panic=1 rootfstype=ext4 rw" \
    -drive "file=${RPI_FS},index=0,media=disk,format=raw" \
    -net user,hostfwd=tcp::5022-:22 -net nic \
    -nographic

Though I don't know why and how this script works. At least, it proves to be working, non-GUI mode (maybe due the the image is 'lite' version). The next step is to work on Raspberry Pi version with GUI mode (next example), and understand how it works. Check :arrow_right:Next article - QEMU for Advanced

Below are some screen shots.

Fig. - Running shell script to launch qemu Raspberry Pi image. qemu create another screen, though showing nothing.

Fig. - Same as above, without qemu screen

Fig. - Boot up sequences show Welcome to Raspbian GNU/Linux 10 (buster)!

Fig. - Prompt for login user id and pass word input. The standard ones are pi and raspberry respectively.

Fig. - Successfully login to Raspbian

Fig. - Shell command uname -a to check Linux kernel version

QEMU Example 5 : `qemu-system-arm` running Raspberry Pi in GUI mode

Following from previous examples and Emuation using Qemu's native raspi2/3 machine, we modified the script by using newer Raspberry Pi images of full version, so we can run startx command in RPi to enter GUI mode.

Here is the script.

#!/bin/sh

QEMU=$(command -v qemu-system-arm)
TMP_DIR=$(pwd)
RPI_KERNEL=kernel-qemu-5.4.51-buster
RPI_KERNEL_FILE=$TMP_DIR/$RPI_KERNEL
PTB=versatile-pb-buster-5.4.51.dtb
PTB_FILE=$TMP_DIR/$PTB
IMAGE_BASE=2020-05-27-raspios-buster-full-armhf
IMAGE=$IMAGE_BASE.zip
IMAGE_FILE=$TMP_DIR/$IMAGE
RPI_FS=$TMP_DIR/$IMAGE_BASE.img

mkdir -p $TMP_DIR

wget https://github.com/dhruvvyas90/qemu-rpi-kernel/blob/master/${RPI_KERNEL}?raw=true \
        -O ${RPI_KERNEL_FILE}

wget https://github.com/dhruvvyas90/qemu-rpi-kernel/raw/master/$PTB \
        -O ${PTB_FILE}

wget http://downloads.raspberrypi.org/raspios_full_armhf/images/raspios_full_armhf-2020-05-28/$IMAGE \
        -O ${IMAGE_FILE}
unzip $IMAGE_FILE -d $TMP_DIR


# $QEMU -kernel ${RPI_KERNEL_FILE} \
#    -cpu arm1176 -m 256 -M versatilepb \
#    -dtb ${PTB_FILE} -no-reboot \
#    -serial stdio -append "root=/dev/sda2 panic=1 rootfstype=ext4 rw" \
#    -drive "file=${RPI_FS},index=0,media=disk,format=raw" \
#    -net user,hostfwd=tcp::5022-:22 -net nic

$QEMU \
  -M versatilepb \
  -cpu arm1176 \
  -m 256 \
  -drive "file=${RPI_FS},if=none,index=0,media=disk,format=raw,id=disk0" \
  -device "virtio-blk-pci,drive=disk0,disable-modern=on,disable-legacy=off" \
  -net "user,hostfwd=tcp::5022-:22" \
  -dtb ${PTB_FILE} \
  -kernel ${RPI_KERNEL_FILE} \
  -append 'root=/dev/vda2 panic=1' \
  -no-reboot

After launch Raspberry Pi successfully (Difference is, this time, it shows on qemu screen, instead of terminal due to removal of qemu -nogrphic option). If qemu screen still shows in command line mode, we can enter Raspberry Pi specific command startx to ente GUI mode.

Below are some screen shots.

Fig. - Launch shell script to download files from internet, including .img, .dtb, kernel. After downloading the image files, and it takes some time to boot up qemu, at that moment I suspect the system was down, but not.

Fig. - qemu launch Raspberry Pi (on another screen)

Fig. - qemu lauch Raspberry Pi into GUI mode

Fig. - Raspberry Pi GUI boot from qemu

Fig. - Raspberry Pi terminal shows the Linux kernel version

Fig. - Raspberry Pi running a game. Unfortunately, it does not support mouse operation. That requires further mouse driver setting at qemu.

This is a separate line to distinguish between Beginner and Avanced

QEMU for Advanced

tags: `2022/09` `qemu` `qemu-user` `qemu-architecture` `qemu-system-architecture`

(2022/9/23) QEMU for advanced, after completing previous examples, moves on to next step to build own version of kernel images for any x86 Linux and Raspberry Pi BIOS images
(latest update on 2022/10/8)

So far, we use the images and kernel from the internet prepared by others. It will not work with combination of newer firmware with older kernels (I tried and failed so many timmes). I am tringing to see how to generate the kernel file from official Raspberry Pi images so we can upgrade the firmware and running under qemu. Also try to understand the parameters of qemu so we can experiments more, like ssh to QEMU RPi. Or we can use the mouse in Raspberry Pi GUI mode.

Before jumping into examples, there are couples of prerequisites.

qemu-system- examples of different options
qemu-system- Block Device Options
qemu-img usage
losetup (my preference) or mount, or nbd to check the contents of img, .iso, and so on images files.

`qemu-system-` examples of different options

I found it is not easy to set the qemu-system- options right to get it working, there are basic options needed to launch qemu-system- successfully. Few examples are listed.

**Using `-kernel kernel` and `-dtb .dtb`**

Option 1 for qemu-system-arm: Use -drive

qemu-system-arm \
  -M versatilepb -cpu arm1176 -m 256 \
  -dtb versatile-pb.dtb \
  -kernel kernel-qemu-4.19.50-buster \
  -append "root=/dev/sda2 panic=1 rootfstype=ext4 rw console=ttyAMA0" \
  -drive "file=2019-09-26-raspbian-buster-lite.img,index=0,media=disk,format=raw" \
  -net user,hostfwd=tcp::2222-:22 -net nic \
  -serial stdio \
  -no-reboot \
  -nographic

Seems console=ttyAMA0 can be removed from -append, no impact.
About the parameter panic=1 (or any number N to replace 1) in -append, it asks qemu-system- to wait 1 (or N) seconds before system to reboot due to kernel panic.
When -M is selected, no need to specify -cpu, as CPU is machine dependent. But when I use -M versatilpb and remove -cpu arm1176, it does not work.
-net user,hostfwd=tcp::2222-:22 -net nic : set up the networking stack and forward the SSH port. Trying to use, but failed -device virtio-net-device,netdev=unet -netdev user,hostfwd=tcp:127.0.0.1:2222-:22,id=unet

Option 2 for qemu-system-arm : Use -hda
Replace the line
-drive "file=2019-09-26-raspbian-buster-lite.img,index=0,media=disk,format=raw"
by
-hda 2019-09-26-raspbian-buster-lite.img.
Check qemu-system- : QEMU Block Device Options for more detail explanation.

Option 3 for qemu-system-arm : Use .qcow2 as the image file.

# create an empty .qcow2
qemu-img create -f qcow2 disk.qcow2 20G

# Convert OS image to .qcow2 format
qemu-img convert -f raw -O qcow2 2019-09-26-raspbian-buster-lite.img disk.qcow2

Then replace the line
-drive "file=2019-09-26-raspbian-buster-lite.img,index=0,media=disk,format=raw"
by
-hda disk.qcow2

Option 4 for qemu-system-arm : Add -blockdev option

It works though not sure if improves or not. However, need to input the file name in both -drive and -blockdev, or it reports error.

-drive file=2019-09-26-raspbian-buster-lite.img,index=0,media=disk,id=hd0,format=raw \
-blockdev driver=raw,node-name=disk,file.driver=file,file.filename=2019-09-26-raspbian-buster-lite.img \

Option 5 for qemu-system-arm Use root=/dev/mmcblk0p2
See below QEMU Advanced Example 2 : Build Raspberry Pi image from downloaded image and buildroot for kernel and .dtb for 32 bit ARM

Option 6 for qemu-system-arm : Add -device virtio-blk-device,drive=hd0 option when -M virtio is selected. It won't work with -M versstilepb.

qemu-system-arm: -device virtio-blk-device,drive=hd0: No 'virtio-bus' bus found for device 'virtio-blk-device'

Check below option 11 for example. The reason why it failed with -M versatilepb is explained in this article stackoverflow - Enabling virtio_blk_device for qemu.

virtio-blk-device is a VirtIO device that relies solely on memory-mapped IO (MMIO) and not on the PCI-bus. This does not work with Qemu's default machine type pc-i440fx-X.Y, or at least not out of the box.

Similarly, it failed for -M versatilepb to support -device virtio-net-device,netdev=unet with error message below.

qemu-system-arm: -device virtio-net-device,netdev=unet: No 'virtio-bus' bus found for device 'virtio-net-device'

Using `-kernel vmlinuz` and `-initrd initrd.gz` or `initrd initrd.img`

Option 11 for qemu-system-arm

wget https://cdimage.debian.org/cdimage/archive/10.8.0/armhf/iso-cd/debian-10.8.0-armhf-xfce-CD-1.iso
sudo mkdir /mnt/rpi32
sudo mount debian-10.8.0-armhf-xfce-CD-1.iso /mnt/rpi32
cp /mnt/rpi32/install.ahf/vmlinuz .
cp /mnt/rpi32/install.ahf/initrd.gz .
sudo umount /mnt/rpi32
qemu-img create disk.qcow2 10G
qemu-system-arm \
  -M virt -cpu cortex-a15 -smp 2 -m 1024 \
  -initrd initrd.gz \
  -kernel vmlinuz \
  -drive if=none,file=disk.qcow2,id=hd0,format=raw \
  -device virtio-blk-device,drive=hd0 \
  -drive if=none,file=debian-10.8.0-armhf-xfce-CD-1.iso,id=cdrom,media=cdrom \
  -device virtio-scsi-device \
  -device scsi-cd,drive=cdrom \
  -nographic

Option 12 for qemu-system-aarch64

qemu-system-aarch64 \
    -M virt -cpu cortex-a57 -smp 2 -m 1024 \
    -initrd initrd.img \
    -kernel vmlinuz \
    -append "root=/dev/sda2 console=ttyAMA0" \
    -device virtio-scsi-device \
    -blockdev qcow2,node-name=hd0,file.driver=file,file.filename=disk.qcow2 \
    -device scsi-hd,drive=hd0 \
    -device virtio-net-device,netdev=unet \
    -netdev user,hostfwd=tcp:127.0.0.1:2222-:22,id=unet \
    -nographic

qemu-system-aarch64 -M virt -cpu cortex-a57 -smp 2 -m 1024 -nographic : run the ARM64 virtual platform emulator with 1GB RAM (or can input as -m 1G) and 2 Cortex-A57 cores with no GUI support.
-drive if=none,file=ubuntu-16.04-server-cloudimg-arm64-uefi1.img,id=hd0 : use the Ubuntu image file
-device virtio-blk-device,drive=hd0 : mount drive from above as a block device
-drive file=user-data.img,format=raw : use the configuration data image file

qemu-system- : QEMU Block Device Options

Check QEMU document page 34 about Block device options

The QEMU block device handling options have a long history and have gone through several iterations as the featureset and complexity of the block layer have grown. Many online guides to QEMU often reference older and deprecatedoptions, which can lead to confusion. The recommended modern way to describe disks is to use a combination of -device to specify the hardware device and -blockdev to describe the backend. The device defines what the guest sees and the backend describes how QEMU handles the data.

qemu-img : QEMU image handling

qemu can support many formats, qcow2 format is recommended.
.qcow is a file format for disk image files used by QEMU. It stands for "QEMU Copy On Write" and uses a disk storage optimization strategy that delays allocation of storage until it is actually needed. Files in qcow format can contain a variety of disk images which are generally associated with specific guest operating systems. Three versions of the format exist: qcow, qcow2 and qcow3. From wiki-qcow

# create an empty .qcow2
qemu-img create -f qcow2 disk.qcow2 20G

# Convert OS image to .qcow2 format
qemu-img convert -f raw -O qcow2 raspberrypi.img disk.qcow2

losetup: Loop back

cd ~
mkdir mnt
mkdir mnt/fat32
mkdir mnt/ext4
# Download Raspberry Pi image first
sudo losetup -f --show -P <path-to-rpi-image-file>
# Mount on mnt/fat32 and mnt/ext4
# sudo mount /dev/loop<no>p1 mnt/fat32 or mnt/ext4
# normally, it is loop0
# fat32 stores the kernel and dtb files
sudo mount /dev/loop0p1 mnt/fat32
# ext4 stores the Raspberry Pi OS
sudo mount /dev/loop0p2 mnt/ext4
sudo losetup -d /dev/loop<no>

Check Native emulation of Rpi2/3 using Qemu's Raspi2/3 machine, and Rasbperry Pi Linux kernel for more details

mount for `.img` and `.iso`

mount for .img
Check the content of OS images by mounting images to Linux drives.

unzip <image-file>.zip
fdisk -l 2019-09-26-raspbian-buster.img

You should see something like this:

Disk 2019-09-26-raspbian-buster.img: 3.6 GiB, 3829399552 bytes, 7479296 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xd9b3f436

Device                          Boot  Start     End Sectors  Size Id Type
2019-09-26-raspbian-buster.img1        8192  532479  524288  256M  c W95 FAT32 (
2019-09-26-raspbian-buster.img2      532480 7479295 6946816  3.3G 83 Linux

You see that the filesystem (.img2) starts at sector 532480. Now take that value and multiply it by 512, in this case it’s 512 * 532480 = 272629760 bytes. Use this value as an offset in the following command:

# others might use #sudo `mkdir /mnt/ext4` more often, which both can serve the same purpose
mkdir mnt/ext4
sudo mount -v -o offset=272629760 -t ext4 2019-09-26-raspbian-buster.img mnt/ext4[sudo] password for kernel-dev: 
# output from the system
mount: /dev/loop0 mounted on /home/kernel-dev/myworks/qemu-arm32/mnt/ext4.

Similarly for .img1, which contains the kernel and .dtb.

sudo mount -v -o offset=4194304 -t vfat 2019-09-26-raspbian-buster.img mnt/fat32
# output from the system
mount: /dev/loop1 mounted on /home/kernel-dev/myworks/qemu-arm32/mnt/fat32.

Read Raspberry Pi on QEMU for more details

mount for .iso

wget https://cdimage.debian.org/cdimage/archive/10.8.0/armhf/iso-cd/debian-10.8.0-armhf-xfce-CD-1.iso
sudo mkdir /mnt/rpi32
sudo mount debian-10.8.0-armhf-xfce-CD-1.iso /mnt/rpi32

nbd:

# Download images and create virtual disk
wget http://ftp.debian.org/debian/dists/buster/main/installer-arm64/current/images/netboot/debian-installer/arm64/initrd.gz
wget http://ftp.debian.org/debian/dists/buster/main/installer-arm64/current/images/netboot/debian-installer/arm64/linux

qemu-img create -f qcow2 disk.qcow2 20G

# nbd
sudo apt install nbd-client qemu qemu-utils
sudo modprobe nbd max_part=8
sudo qemu-nbd --connect=/dev/nbd0 disk.qcow2
sudo mount /dev/nbd0p1 /media/qcow
sudo cp initrd.img-4.19.0-16-arm64 vmlinuz-4.19.0-16-arm64 ../qemu/.
sync
sudo umount /dev/nbd0p1
sudo nbd-client -d /dev/nbd0

Check Building a Debian Buster QEMU image for AARCH64 for more details

QEMU Advanced Example 1 : Run QEMU Debian image by using kernel and initrd.gz from ftp.debian.org for 64 bit ARM aarch64

I succeeded in following this article Building a Debian Buster QEMU image for AARCH64, after upgrading to qemu version 5.0.0 on my Ubuntu 16.04 by source compiling.

This article serves my intention well to use any downloadable Linux image and extract initrd and vmlinuz from that image required for qemu. So it is a 2-step approach.

1st step is to get the Debian image for aarch64

wget http://ftp.debian.org/debian/dists/buster/main/installer-arm64/current/images/netboot/debian-installer/arm64/initrd.gz
wget http://ftp.debian.org/debian/dists/buster/main/installer-arm64/current/images/netboot/debian-installer/arm64/linux

2nd step is to create a 20GB file as the disk for Debian ISO image to install on It is similar to select the HDD drive which we are going to install Debian ISO image and boot from. So the `disk.qcow2` content can be empty for now. It is recommended to use the format of `.qcow2` Copy-On-Write format.

qemu-img create -f qcow2 disk.qcow2 20G

3rd step is to install Debian image on `qemu-system-aarch64` by using the following command.

qemu-system-aarch64 -smp 2 -M virt -cpu cortex-a57 -m 1G \
    -initrd initrd.gz -kernel linux \
    -append "root=/dev/ram console=ttyAMA0" \
    -device virtio-scsi-device \
    -blockdev qcow2,node-name=hd0,file.driver=file,file.filename=disk.qcow2 \
    -device scsi-hd,drive=hd0 \
    -netdev user,id=unet -device virtio-net-device,netdev=unet \
    -nographic

After successfully launching the Debian image, it starts installation which takes really some time. (Remember the root password you key in, and user id and password you create.) After installation complete, the system prompt

                                                                                
   ┌─────────────────┤ [!] Continue without boot loader ├──────────────────┐    
   │                                                                       │    
   │                       No boot loader installed                        │    
   │ No boot loader has been installed, either because you chose not to or │    
   │ because your specific architecture doesn't support a boot loader yet. │    
   │                                                                       │    
   │ You will need to boot manually with the /vmlinuz kernel on partition  │    
   │ /dev/sda1 and root=/dev/sda2 passed as a kernel argument.             │    
   │                                                                       │    
   │                              <Continue>                               │    
   │                                                                       │    
   └───────────────────────────────────────────────────────────────────────┘

Click <Continue> (When I tried another example by installing Ubuntu (not Debian), it reports an error message. This error can be ignore and skip the step of installing boot loader, then complete rest of the installation.)

                                                                          
                                                                                
   ┌───────────────────┤ [!!] Finish the installation ├────────────────────┐    
   │                                                                       │    
  ┌│                         Installation complete                         │    
  ││ Installation is complete, so it is time to boot into your new system. │    
  ││ Make sure to remove the installation media, so that you boot into the │    
  ││ new system rather than restarting the installation.                   │    
  ││                                                                       │    
  ││     <Go Back>                                          <Continue>     │    
  └│                                                                       │    
   └───────────────────────────────────────────────────────────────────────┘

and now, we can press CTRL-a, then x to quie qemu.

4th step is to extract the initrd.img and vmlinuz files from the qcow2 disk file The easiest way to do this is to use nbd in a Linux environment (the host Linux I have been using).

sudo apt install nbd-client qemu qemu-utils
sudo modprobe nbd max_part=8
sudo qemu-nbd --connect=/dev/nbd0 disk.qcow2
sudo mkdir -p /media/qcow
sudo mount /dev/nbd0p1 /media/qcow
# make sure the following file names are correct versions
cp /media/qcow/initrd.img-4.19.0-16-arm64 .
cp /media/qcow/vmlinuz-4.19.0-16-arm64 .
sync
sudo umount /dev/nbd0p1
sudo nbd-client -d /dev/nbd0

5th step is to run Debian which we installed on `disk.qcow2` with `initrd` and `vmlinuz`

qemu-system-aarch64 -smp 2 -M virt -cpu cortex-a57 -m 1G \
  -initrd initrd.img-4.19.0-16-arm64 \
  -kernel vmlinuz-4.19.0-16-arm64 \
  -append "root=/dev/sda2 console=ttyAMA0" \
  -device virtio-scsi-device \
  -blockdev qcow2,node-name=hd0,file.driver=file,file.filename=disk.qcow2 \
  -device scsi-hd,drive=hd0 \
  -netdev user,hostfwd=tcp:127.0.0.1:2222-:22,id=unet -device virtio-net-device,netdev=unet \
  -nographic

I add hostfwd=tcp:127.0.0.1:2222-:22, (not in original article) after -netdev user to allow host to ssh via -p 2222

Now we can run Debian on qemu of ARM 64bit aarch64 architecture for any Debian images in the future. Or in theory, for all architectures.

QEMU Advanced Example 2 : Build Raspberry Pi image from downloaded image and buildroot for kernel and .dtb for 32 bit ARM

A : This example combines the following articles

ARM64 vanilla kernel in QEMU to learn how to install buildroot and how to make.
Raspberry Pi Documentation - The Linux kernel to learn how to customize buidroot make bcmxxxx_defconfig for different Raspberry Pi boards.
Raspbian on QEMU with ARMv7 to learn the qemu command line and options with kernel zImage and .dtb from buildroot, and .img from downloads.raspberrypi.com

Below are the commands for this example:

# Reference - https://kasiviswanathanblog.wordpress.com/2020/05/03/arm64-vanilla-kernel-in-qemu/
git clone git://git.buildroot.net/buildroot
cd buildroot/

# Reference - https://www.raspberrypi.com/documentation/computers/linux_kernel.html#using-menuconfig
# Using below buildroot commands
cd buildroot/
# list the pre-defined configs in buildroot
make list-defconfigs
# buildroot for arm under qemu
make clean
make qemu_arm_vexpress_defconfig         # can try other qemu arm defconfig
make menuconfig                          # Exit to save
make 2>&1 | tee build.log
# Output files are located in the directory output/images/

# Reference - https://github.com/paulden/raspbian-on-qemu-armv7
# Download .img from raspberrypi.org
wget http://downloads.raspberrypi.org/raspbian_full/images/raspbian_full-2020-02-14/2020-02-13-raspbian-buster-full.zip
unzip 2020-02-13-raspbian-buster-full.zip

# qemu help to check the machines that qemu supports
qemu-system-arm -machine help

# command to execute qemu with options
qemu-system-arm \
        -M vexpress-a9 \
        -m 1G \
        -kernel output/images/zImage \
        -dtb output/images/vexpress-v2p-ca9.dtb \
        -sd 2020-02-13-raspbian-buster-full.img \
        -append "console=ttyAMA0,115200 root=/dev/mmcblk0p2" \
        -serial stdio \
        -net nic -net user,hostfwd=tcp::2222-:22


# The following option from Example 2 does not work on this case, not sure why.
    -drive "file=2020-02-13-raspbian-buster-full.img,index=0,media=disk,format=raw" \
    -append "root=/dev/sda1 panic=1 rootfstype=ext4 rw console=ttyAMA0,115200"

QEMU Advanced Example 3 : Run QEMU Raspberry Pi image from download and cross compiling Raspberry Pi Linux source to get kernel and .dtb for 32 bit ARM (still failed)

A : This example follows Raspberry Pi Documentation - The Linux kernel

Install Required Dependencies and Toolchain, in case not yet done

# To build the sources for cross-compilation, make sure you have the dependencies needed on your machine by executing:
sudo apt install git bc bison flex libssl-dev make libc6-dev libncurses5-dev

# Install the 32-bit Toolchain for a 32-bit Kernel
sudo apt install crossbuild-essential-armhf

#Install the 64-bit Toolchain for a 64-bit Kernel
sudo apt install crossbuild-essential-arm64

Get the Kernel Sources

# To download the minimal source tree for the current branch, run:
git clone --depth=1 https://github.com/raspberrypi/linux
cd linux

Build sources

Enter the following commands to build the sources and Device Tree files. There are several config's for ARM 32 bits or 64 bits, and different versions of kernels (refer to above article for more details). I pick this one of Pi 3+, which I am familiar with.


# For Raspberry Pi 2, 3, 3+ and Zero 2 W, and Raspberry Pi Compute Modules 3 and 3+:

KERNEL=kernel7
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- bcm2711_defconfig

Build with Configs

Note
To speed up compilation on multiprocessor systems, and get some improvement on single processor ones, use make -j n, where n is the number of processors. You can use the nproc command, like make -j $(nproc) to see how many processors you have. Alternatively, feel free to experiment and see what works!

# For all 32-bit Builds
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- zImage modules dtbs

The file locations are scattered across serveral directories:

vmlinux : linux/vmlinux/
Image / zImage : linux/arch/arm/boot/
dtb : linux/arch/arm/boot/dts/bcm2709-rpi-2-b.dtb

Launch qemu with the options from previous example.

# Download Raspberry Pi image 
wget http://downloads.raspberrypi.org/raspbian/images/raspbian-2019-09-30/2019-09-26-raspbian-buster.zip
unzip 2019-09-26-raspbian-buster.zip

qemu-system-arm \
    -M raspi2 \
    -m 1G \
    -kernel linux/arch/arm/boot/zImage \
    -dtb linux/arch/arm/boot/dts/bcm2709-rpi-2-b.dtb \
    -sd 2019-09-26-raspbian-buster.img \
    -append "root=/dev/mmcblk0p2 panic=1 rootfstype=ext4 rw" \
    -serial stdio \
    -net user,hostfwd=tcp::5022-:22 -net nic

Unfortunately, it failed to boot, and stopped at

Rebooting in 1 seconds..
Reboot failed -- system halted

There are some information mentioned in stackexchange - Kernel and QEMU : Unable to mount root fs error, which maybe worthwhile to spend some time to look into it in more details.

This error message means that the kernel is unable to mount the partition you requested to be /. This would happen for example if you gave it an empty disk image (my hunch is this is your case) - the kernel in the VM sees an unpartitioned drive, there is no /dev/sda1 just /dev/sda. To overcome this follow the instructions in the guide you have used - download a bootable ISO image and use it to install system into the VM image. When raw disk image is used, it can be directly partitioned with utilities like gdisk, fdisk or parted.

Another possibility is, that there you are trying to mount a filesystem for which the kernel doesn't have a driver. This usually happens when one uses a kernel, that has most drivers in loadable modules on initrd and the initrd isn't loaded (hence the kernel lacks the ability to understand the particular filesystem).

If you made a full installation of Debian, I would expect it having installed the kernel as well. Hence it should be possible to bring up the VM without the -kernel option at all, since the VM BIOS should be able to boot the installed system right away as a real BIOS would do - by loading bootloader from MBR (or EFI partition, although the UEFI support in Qemu/KVM is still rather new AFAIK).

QEMU Advanced Example 4 : Run QEMU Debian image using .iso only (to extract `vmlinuz` and `initrd.gz`) for 32 bit ARM

Follow this article Easy way to run qemu Debian OS - 使用qemu-system-arm安裝系統(armhf) to run qemu from download image.

wget https://cdimage.debian.org/cdimage/archive/10.8.0/armhf/iso-cd/debian-10.8.0-armhf-xfce-CD-1.iso
sudo mkdir /mnt/rpi32
sudo mount debian-10.8.0-armhf-xfce-CD-1.iso /mnt/rpi32
cp /mnt/rpi32/install.ahf/vmlinuz .
cp /mnt/rpi32/install.ahf/initrd.gz .
sudo umount /mnt/rpi32
qemu-img create disk.qcow2 10G
qemu-system-arm \
  -m 1024 -cpu cortex-a15 -smp 2 -M virt \
  -kernel vmlinuz \
  -initrd initrd.gz \
  -drive if=none,file=disk.qcow2,id=hd0,format=raw \
  -device virtio-blk-device,drive=hd0 \
  -drive if=none,file=debian-10.8.0-armhf-xfce-CD-1.iso,id=cdrom,media=cdrom \
  -device virtio-scsi-device \
  -device scsi-cd,drive=cdrom \
  -nographic

References

qemu Documentation
qemu Documentation Index
qemu Documentation PDF - latest
Suse - Running Virtual Machines with qemu-system-ARCH REPORT DOCUMENTATION - Using Devices in QEMU : Quite some useful tips to run QEMU.
qemu System Invocation
QEMU related articles and issues
- How to emulate a Raspberry Pi on your PC
- Running Raw Linux Kernel in QEMU - initial ramdisk
- stackexchange - no network interface in QEMU
- stackoverflow - Difference between -net user and -net nic in qemu
- Emuation using Qemu's native raspi2/3 machine : Best article about qemu application I learned so far.
- Using QEMU to emulate a Raspberry Pi
  - sudo qemu-system-arm: you need to run QEMU as root
  - -kernel: this is the path to the QEMU kernel we downloaded in the previous step
  - -append: here we are providing the boot args direct to the kernel, telling it where to find it's root filesytem and what type it is
  - -hda: here we're attaching the disk image itself
  - -cpu/-m: this sets the CPU type and RAM limit to match a Raspberry Pi
  - -M: this sets the machine we are emulating. versatilepb is the 'ARM Versatile/PB' machine
  - -no-reboot: just tells QEMU to exit rather than rebooting the machine
  - -serial: redirects the machine's virtual serial port to our host's stdio
    -net: this configures the machine's network stack to attach a NIC, use the user-mode stack, connect the host's vnet0 TAP device to the new NIC and don't use config scripts.
QEMU examples programs from internet
- Example 1 - Building a Debian Buster QEMU image for AARCH64 : Working perfectly
- Example 2 - Raspbian on QEMU with ARMv7 : working fine.
- Example 3 - Raspberry Pi Linux kernel : It works fine to build Raspberry Pi kernel from Linux source code. It applied to SD card, iso QEMU.
- Example 4 - Debian on ARM via QEMU : Tried and failed
- Example 5 - Debian on QEMU-emulated ARM-64 aarch64 : Tried and failed
- Example 6 - Emuation using Qemu's native raspi2/3 machine : Best article about qemu application I learned so far.
  - Step by step guide to extract dtb files and kernel image from img file : This article should help explain how to get dtb and kernel image from img file, instead of counting on other works, which is not portable. This github provide tools and documents to generate kernel images for qemu. It is worthwhile to spend time to study more here.
- Example 7 - Emulating ARM64 Raspberry Pi Image using QEMU : A video clearly explain the procedure step by step. Though I failed to reproduce.
- Example 8 - RASPBERRY PI ON QEMU : It is similar to the video of above, but explaned in text, iso video. And I failed to reproduce.
- Example 9 - 使用qemu-system-arm安裝系統(armhf)
- Example 10 - Get an ARM64 version of Ubuntu running in the QEMU emulator on Windows 10. : This article explains the meaning of parameters to qemu.

qemu-system-aarch64 -m 2048 -cpu cortex-a72 -smp 4 -M virt -nographic -bios QEMU_EFI.fd -drive if=none,file=ubuntu-16.04-server-cloudimg-arm64-uefi1.img,id=hd0 -device virtio-blk-device,drive=hd0 -drive file=user-data.img,format=raw -device virtio-net-device,netdev=net0 -netdev user,hostfwd=tcp:127.0.0.1:2222-:22,id=net0

qemu-system-aarch64 -m 2048 -cpu cortex-a72 -smp 4 -M virt -nographic - run the ARM64 virtual platform emulator with 2GB RAM and 4 Cortex-A72 cores with no GUI support.
-bios QEMU_EFI.fd - use the firmware downloaded above.
-drive if=none,file=ubuntu-16.04-server-cloudimg-arm64-uefi1.img,id=hd0 - use the Ubuntu image file
-device virtio-blk-device,drive=hd0 - mount drive from above as a block device
-drive file=user-data.img,format=raw - use the configuration data image file
-device virtio-net-device,netdev=net0 - create a virtual network device
-netdev user,hostfwd=tcp:127.0.0.1:2222-:22,id=net0 - set up the networking stack and forward the SSH port

QEMU Q&A
- qemu-system-x86_64: drive with bus=0, unit=0 (index=0) exists 錯誤原因： When encoutered error of qemu-system-i386: -m 16: drive with bus=0, unit=0 (index=0) exists, fix it with 可以把參數重新用手動輸入的方式敲，不要直接複製，因爲複製過程中可能複製了一些我們沒注意到的字符導致程序不識別；
- QEMU emulating Raspberry Pi using verstailepb machine?
  
  I download the Rapsberry OS
  image
  https://downloads.raspberrypi.org/raspios_armhf/images/raspios_armhf-2021-01-12/2021-01-11-raspios-buster-armhf.zip
  
  the kernel kernel-qemu-4.19.50-buster and versatile-pb-buster.dtb
  from https://github.com/dhruvvyas90/qemu-rpi-kernel and run with
  version 5.2.0:
  
  $ qemu-system-arm -dtb ./versatile-pb-buster.dtb -kernel ./kernel-qemu-
  4.19.50-buster -cpu arm1176 -m 256 -M versatilepb
  - You are attempting to boot a generic kernel with a versatile dtb on a verstailepb machine. The site you link to even has pointers towards "native-emulation" for using the -M raspi2/3 models.
    The documentation also lists the current status of the emulation:
    https://qemu.readthedocs.io/en/latest/system/arm/raspi.html
Cross compiler
- Cross compiling for arm or aarch64 on Debian or Ubuntu
Raspberry Pi related
- Raspberry Pi Images Downloads
- Michael Stapelberg’s Debian Blog : It seems to be a maintainer of Raspberry Pi image, and looking for someone can take over, or becomes unmaintained.
- Script to simulate Raspberry Pi with QEMU
Linux Q&A
Buildroot
Linux boot sequence
- Linux Bootstrap (開機載入)
- Step by step guide to extract dtb files and kernel image from img file
- Raspberry Pi
  - Stackexchange - Running Raspberry PI buildroot image in qemu : After few days of googling around a lot i found the answer to my issue. The Kernel image zImage I use in the terminal command which is built by the buildroot make process is not compatible with the QEMU. To solve that I had to compile my own QEMU compatible raspberry pi kernel. I followed this link to compile my own kernel. raspberry pi cross-compile kernel then when I used QEMU to run a OS there was another kernel panic saying VFS mount failed. When I read the terminal outputs before that, it says not enough space on the media. So expanded the scared.img file to 1G and ran the same terminal command. Now the QEMU is running perfectly
  - RaspberryPi.com Linux kernel :
RaspberryPi.org Images Downloads
Cross compiler
- Cross compiling for arm or aarch64 on Debian or Ubuntu

:arrow_left:Previous article - back to marconi's blog
:arrow_right:Next article - QEMU simulation ith hardware board
:arrow_up:back to marconi's blog

QEMU for Beginner and Advanced

tags: 2022/09 qemu qemu-user qemu-architecture qemu-system-architecture

What is qemu?

QEMU Installation

QEMU Example 1 : qemu-x86_64 and qemu-i386

Running hello world on Linux native environmemt

Running hello world on qemu

QEMU Example 2 : qemu-aarch64

Install ARM toolchain

QEMU Example 3 : qemu-system-i386

QEMU Example 4 : qemu-system-arm running Raspberry Pi in command mode

QEMU Example 5 : qemu-system-arm running Raspberry Pi in GUI mode

QEMU for Advanced

tags: 2022/09 qemu qemu-user qemu-architecture qemu-system-architecture

qemu-system- examples of different options

Using -kernel kernel* and -dtb *.dtb

Using -kernel vmlinuz and -initrd initrd.gz or initrd initrd.img

qemu-system- : QEMU Block Device Options

qemu-img : QEMU image handling

losetup: Loop back

mount for .img and .iso

nbd:

QEMU Advanced Example 1 : Run QEMU Debian image by using kernel and initrd.gz from ftp.debian.org for 64 bit ARM aarch64

1st step is to get the Debian image for aarch64

2nd step is to create a 20GB file as the disk for Debian ISO image to install on It is similar to select the HDD drive which we are going to install Debian ISO image and boot from. So the disk.qcow2 content can be empty for now. It is recommended to use the format of .qcow2 Copy-On-Write format.

3rd step is to install Debian image on qemu-system-aarch64 by using the following command.

4th step is to extract the initrd.img and vmlinuz files from the qcow2 disk file The easiest way to do this is to use nbd in a Linux environment (the host Linux I have been using).

5th step is to run Debian which we installed on disk.qcow2 with initrd and vmlinuz

QEMU Advanced Example 2 : Build Raspberry Pi image from downloaded image and buildroot for kernel and .dtb for 32 bit ARM

QEMU Advanced Example 3 : Run QEMU Raspberry Pi image from download and cross compiling Raspberry Pi Linux source to get kernel and .dtb for 32 bit ARM (still failed)

Install Required Dependencies and Toolchain, in case not yet done

Get the Kernel Sources

Build sources

Build with Configs

QEMU Advanced Example 4 : Run QEMU Debian image using .iso only (to extract vmlinuz and initrd.gz) for 32 bit ARM

References

Read more

Q&A - Linux related

Midjourney

Q&A - Advanced Linux and C Language related

讀書心得 : 操作系統原型 - xv6 分析與實踐 羅秋明 著

tags: `2022/09` `qemu` `qemu-user` `qemu-architecture` `qemu-system-architecture`

QEMU Example 1 : `qemu-x86_64` and `qemu-i386`

Running `hello world` on Linux native environmemt

Running `hello world` on qemu

QEMU Example 2 : `qemu-aarch64`

QEMU Example 3 : `qemu-system-i386`

QEMU Example 4 : `qemu-system-arm` running Raspberry Pi in command mode

QEMU Example 5 : `qemu-system-arm` running Raspberry Pi in GUI mode

tags: `2022/09` `qemu` `qemu-user` `qemu-architecture` `qemu-system-architecture`

`qemu-system-` examples of different options

**Using `-kernel kernel` and `-dtb .dtb`**

Using `-kernel vmlinuz` and `-initrd initrd.gz` or `initrd initrd.img`

mount for `.img` and `.iso`

3rd step is to install Debian image on `qemu-system-aarch64` by using the following command.

5th step is to run Debian which we installed on `disk.qcow2` with `initrd` and `vmlinuz`

QEMU Advanced Example 4 : Run QEMU Debian image using .iso only (to extract `vmlinuz` and `initrd.gz`) for 32 bit ARM

讀書心得 : 操作系統原型 - xv6 分析與實踐羅秋明著