### 第2,3,4,5週：ARM64 Linux on QEMU - Linux from scratch

**內容**：
- 安裝 QEMU、ARM64工具鏈 (`aarch64-linux-gnu-gcc`)
- Linux從開機→Kernel載入→Userland過程
- 學習裝置樹 (device-tree)
- 編譯並啟動簡化的ARM64 Linux系統
- `make menuconfig` 與 Kernel 設定
- rootfs, initramfs, init system 概念
- Lab：使用 QEMU 啟動一個 minimal Linux image

---

#### Prerequisite

$ git clone [e-mail]:buildroot/buildroot
$ cd buildroot
$ git checkout 2025.08
$ make qemu_aarch64_ebbr_defconfig
$ time make

* Why EBBR ?
* What are firmware and bootloader ?
* What are exception levels ?

#### Embedded Linux Target Environment

| **Category** | **Linux From Scratch (BusyBox)** | **Buildroot** | **Distro (Debian/Ubuntu)** |
|---------------|----------------------------------|----------------|-----------------------------|
| **Overview** | Manual assembly of a minimal Linux system using BusyBox, custom kernel, and manually built rootfs. | Automated build system that generates complete embedded Linux images (kernel + rootfs). | Pre-built, general-purpose distribution adapted for embedded use. |
| **Complexity** | Very high (everything built manually). | Moderate (automated but heavily configurable). | Low (ready-to-use packages and tools). |
| **Build Time** | Long — full manual compile of all components. | Medium — builds automatically after configuration. | Short — minimal rebuilding; uses precompiled binaries. |
| **Customization** | Maximum — full control over every component and configuration. | High — menuconfig allows easy selection/removal of packages. | Limited — full-stack OS; removing or modifying system components is complex. |
| **Maintenance** | Hard — you must track updates and dependencies yourself. | Easier — Buildroot manages updates and reproducible builds. | Easy — package manager (APT) handles updates and dependencies. |
| **Filesystem Size** | Very small (<10 MB typical with BusyBox). | Small to medium (10–100 MB depending on packages). | Large (hundreds of MB to several GB). |
| **Performance** | Excellent — minimal overhead. | Good — lean system optimized for target. | Moderate — more services and daemons running. |
| **Package Management** | None — everything is static/custom-built. | Optional — static builds or custom overlay scripts. | Full dpkg/APT system. |
| **Target Devices** | Very small embedded systems, microcontrollers with MMU. | Typical embedded boards (ARM, MIPS, RISC-V). | Powerful SBCs (Raspberry Pi, Jetson, etc.) with large storage. |
| **Learning Curve** | Steep — requires deep kernel, boot, and cross-compilation knowledge. | Moderate — good balance of control and automation. | Gentle — feels like desktop Linux. |
| **Use Case Examples** | IoT sensors, bootloaders, minimal diagnostic systems. | Routers, industrial controllers, embedded products. | Full-featured devices like kiosks, gateways, edge computing. |

#### Build a Minimal Linux Kernel

The mainline Linux kernels are published on kernel.org


$ git clone [e-mail]:torvalds/linux
$ cd linux
$ git checkout v6.18
$ export PATH=$TOOLCHAIN_BIN:$PATH
$ export CROSS_COMPILE=aarch64-linux-
$ make ARCH=arm64 defconfig
$ make ARCH=arm64 -j20

qemu-system-aarch64 \
      -M virt,secure=on,acpi=off \
      -bios flash.bin \
      -cpu cortex-a53 \
      -device virtio-net-device,netdev=eth0 \
      -device virtio-rng-device,rng=rng0 \
      -m 4G \
      -netdev user,id=eth0,hostfwd=tcp::2222-:22,tftp=<your_host_path> \
      -nographic \
      -object rng-random,filename=/dev/urandom,id=rng0 \
      -rtc base=utc,clock=host \
      -smp 4 -serial pty -S -s

Access serial port:

export PTS_NUM=5 && minicom -D /dev/pts/${PTS_NUM} -C /tmp/.pts-${PTS_NUM}-$(date +%Y-%m-%d)

> How to exit minicom: `<Ctrl+a> + x`

Connect QEMU's GDB server from host:

${CROSS_COMPILE}gdb -quiet -ex "target remote 0:1234"

#### Build a Minimal Root File System

$ git clone [e-mail]:mirror/busybox
$ cd busybox
$ git checkout 371fe9f71d445d18be28c82a2a6d82115c8af19d
$ git checkout -b dev
$ export BUSYBOX_PATCH_DIR=/path/to/buildroot/package/busybox/
$ git am $BUSYBOX_PATCH_DIR/*patch
$ make defconfig
$ make -j
$ file busybox
bin/busybox: ELF 64-bit LSB executable, ARM aarch64, version 1 (GNU/Linux), statically linked, for GNU/Linux 3.7.0, stripped

$ make CONFIG_PREFIX=./_install install
$ cd _install
$ mkdir -p proc sys dev etc tmp var mnt/root
$ mkdir -p etc/init.d
$ cat > etc/init.d/rcS <<'EOF'
#!/bin/sh
mount -t proc none /proc
mount -t sysfs none /sys
mount -t tmpfs tmpfs /tmp
echo ""
echo " /\$\$   /\$\$           /\$\$ /\$\$                 /\$\$       /\$\$                              "
echo "| \$\$  | \$\$          | \$\$| \$\$                | \$\$      |__/                              "
echo "| \$\$  | \$\$  /\$\$\$\$\$\$ | \$\$| \$\$  /\$\$\$\$\$\$       | \$\$       /\$\$ /\$\$\$\$\$\$\$  /\$\$   /\$\$ /\$\$   /\$\$"
echo "| \$\$\$\$\$\$\$\$ /\$\$__  \$\$| \$\$| \$\$ /\$\$__  \$\$      | \$\$      | \$\$| \$\$__  \$\$| \$\$  | \$\$|  \$\$ /\$\$/"
echo "| \$\$__  \$\$| \$\$\$\$\$\$\$\$| \$\$| \$\$| \$\$  \ \$\$      | \$\$      | \$\$| \$\$  \ \$\$| \$\$  | \$\$ \  \$\$\$\$/ "
echo "| \$\$  | \$\$| \$\$_____/| \$\$| \$\$| \$\$  | \$\$      | \$\$      | \$\$| \$\$  | \$\$| \$\$  | \$\$  >\$\$  \$\$ "
echo "| \$\$  | \$\$|  \$\$\$\$\$\$\$| \$\$| \$\$|  \$\$\$\$\$\$/      | \$\$\$\$\$\$\$\$| \$\$| \$\$  | \$\$|  \$\$\$\$\$\$/ /\$\$/\  \$\$"
echo "|__/  |__/ \_______/|__/|__/ \______/       |________/|__/|__/  |__/ \______/ |__/  \__/"
echo ""
exec /bin/sh
EOF
$ chmod +x etc/init.d/rcS
$ fakeroot sh -c '
mknod dev/console c 5 1
mknod dev/null c 1 3
find . | cpio -H newc -ov --owner root:root > ../rootfs.cpio
'
$ gzip -9 ../rootfs.cpio

=> dhcp
=> tftp 0x50000000 Image
=> tftp 0x58000000 rootfs.cpio.gz
=> setenv bootargs "console=ttyAMA0 earlycon root=/dev/ram0 rdinit=/sbin/init"
=> booti 0x50000000 0x58000000:$filesize $fdtcontroladdr

## ARM Boot Flow

On an ARMv8‑A SoC, the processor starts in the highest privilege level (EL3) and then gradually hands control to less‑privileged software layers until the operating system and user applications run.

Credit: https://youtu.be/GXFw8SV-51g?t=723

### ROM (BL1) – First Stage Boot Loader (EL3)

* Hard‑wired code inside the chip, executed immediately after reset.
* Initializes only what is strictly necessary (clocks, basic memory, security state).
* Authenticates and loads the next stage from non‑volatile storage (eMMC, NAND, SPI flash, etc.).
* Runs in EL3 with full secure privileges.

### SPL (BL2) – Second Stage / SoC Boot Loader (EL3)

* A small, board‑specific loader stored in flash.
* Performs DDR initialization, basic peripheral setup, and loads the trusted firmware and non‑secure bootloader images into RAM.
* Still executes in EL3, under ROM’s control and security policies.
* Hands off to Trusted Firmware‑A once the system is ready.

### Trusted Firmware‑A (TF‑A, BL31) – EL3 Runtime Services

* Provides the secure monitor and runtime services at EL3.
* Sets up the different Exception Levels (EL3 → EL2 → EL1), configures security state and exception routing.
* Loads and jumps to the non‑secure world: usually a hypervisor at EL2 or directly to the OS kernel at EL1.
* Stays resident to handle secure monitor calls (SMCs) from lower ELs.

### U‑Boot (BL33) – Non‑Secure World Bootloader (EL2 / EL1)

* Loaded as BL33, the non‑secure payload, by TF‑A.
* Runs in the non‑secure world, typically at EL2 (if using a hypervisor setup) or EL1.
* Provides a rich bootloader environment: console, environment variables, networking, storage access, and image loading commands.
* Loads the Linux kernel, device tree, and initramfs (if used) into RAM, sets up boot arguments, and then transfers control to the Linux kernel at EL1.

| Stage / Name | Typical TF‑A ID | Runs At | Role in Boot Flow |
|-----------------------|-----------------|---------|------------------------------------------------------------------------------------|
| ROM (Mask ROM) | BL1 | EL3 | Immutable first code after reset; minimal init, authenticates & loads SPL/BL2. |
| SPL (Secondary Loader)| BL2 | EL3 | Initializes DRAM and basic SoC, loads TF‑A (BL31) and non-secure payload (BL33). |
| TF‑A Runtime Firmware | BL31 | EL3 | Secure monitor and EL3 services; sets up EL2/EL1 and jumps to BL33 (non-secure). |
| U‑Boot (Bootloader) | BL33 | EL2/EL1 | Non-secure payload; full-featured bootloader that loads and boots the Linux kernel.|
| Linux Kernel | — | EL1 | Operating system; initializes drivers, userspace, and runs applications at EL0. |