# Linux 核心專題: Trusted Firmware-A 及 Linux 啟動流程分析 > 執行人: chien1214 > []() :::success :question: 提問清單 * ? ::: ## 任務簡述 [Trusted Firmware](https://www.trustedfirmware.org/) 是 Arm 公司針對 TrustZone 和 [Platform Security Architecture](https://www.arm.com/architecture/security-features/platform-security) 開發的基礎建設,本任務理解 Trusted Firmware-A (TF-A) 及探討 Linux 在 TF-A 的系統啟動流程分析。預計產出: * 從 Arm 第一手材料理解 [Trusted Firmware](https://www.trustedfirmware.org/) 的設計考量和實作規劃,嘗試紀錄和整理 * 以 Trusted Firmware-A (TF-A) 作為切入,探討 Linux 在 TF-A 的系統啟動流程分析 * 以 Raspberry Pi 4B 作為主要實驗平台,搭配硬體 JTAG probe 分析和追蹤 ## TODO: 整理第一手材料 以 [Arm Developer](https://developer.arm.com/) 網站為主要依據 (避免羅列 Arm, Linaro, Pengutronix 以外的技術文件,原始程式碼不在此限),搭配閱讀以下簡報 (和錄影): * [An Introduction to ARM Trusted Firmware](https://www.slideshare.net/linaroorg/arm-trusted-firmareforarmv8alcu13) / [錄影](https://youtu.be/q32BEMMxmfw) * [Technical Overview of Trusted Firmware-A](https://elinux.org/images/0/05/Elc-tfa.pdf) / [錄影](https://youtu.be/0ALBfpjJoC4) * [Trusted Firmware Deep Dive](https://www.linaro.org/app/resources/Connect%20Events/Trusted_Firmware_Deep_Dive_v1.0_.pdf) * [How ARM Systems are Booted: An Introduction to the ARM Boot Flow](https://youtu.be/GXFw8SV-51g) * [ARM Trusted Firmware roadmap and progress](https://www.slideshare.net/linaroorg/lcu14-500-arm-trusted-firmware) / [錄影](https://www.youtube.com/watch?v=je0_-yYgKdc) 解讀針對 Arm 處理器搭配 Trusted Firmware-A (TF-A),Linux 核心的啟動流程、如何確保 TF-A 的作用,並列舉關鍵程式碼及元件。 ## TODO: 以 TF-A 切入,探討 Linux 在 TF-A 的系統啟動流程分析 以 Raspberry Pi 4B 作為主要實驗平台,搭配硬體 JTAG probe 分析和追蹤,不僅列出關鍵程式碼,需要搭配實驗來確認關鍵行為。 ## 流程分析 ## 實驗平台與硬體需求 * [Raspberry Pi 4 Model](https://www.raspberrypi.com/products/raspberry-pi-4-model-b/specifications/) * Broadcom BCM2711 * Quad core Cortex-A72 (ARMv8) 64-bit SoC * 8GB LPDDR4-3200 SDRAM * Host Linux machine * Intel Core i5-4460 * x86_64 * Ubuntu 22.04.2 LTS * USB to UART TTL Cable 4-pin (PL-2303HX) * JTAG debug probe ([J-Link](https://www.segger.com/products/debug-probes/j-link/#comparison)) ## SD card 磁碟分割與格式化 相較於傳統硬碟 (HDD) 與固態硬碟 (SSD), Raspberry Pi 使用 SD card 作為儲存裝置,用以存放系統軟體,包含實驗所需的 bootloader、 firmware 以及作業系統。因此 SD card 需要先進行磁碟分割 (2 partitions),並將其格式化。 * [Partition #1] - boot partition (FAT) * [Partition #2] - rootfs partition (EXT4) 1. 將 SD card 透過讀卡裝置連接上 Host,並找到系統分配給 SD card 的設備節點 (you may also use `lsblk`): ```shell $ cat /proc/partitions major minor #blocks name 8 64 30263296 sdx 8 65 261120 sdx1 8 66 29998080 sdx2 ``` 2. 使用 `fdisk` 分割出實驗所需的兩個磁區 (boot partition & rootfs partition): ```shell $ sudo fdisk /dev/sdx Command (m for help): p Device Boot Start End Sectors Size Id Type /dev/sdx1 * 8192 530431 522240 255M c W95 FAT32 (LBA) /dev/sdx2 530432 60526591 59996160 28.6G 83 Linux ``` 3. 格式化上述兩個分區: ```shell $ sudo mkfs.vfat -n boot /dev/sdx1 $ sudo mkfs.ext4 -L rootfs /dev/sdx2 ``` ## Firmware 更新與寫入 1. 於 [rpi firmware](https://github.com/raspberrypi/firmware) 取得 `/boot` 中的 pre-compiled binaries: ```shell $ git clone --branch stable https://github.com/raspberrypi/firmware ``` 2. 掛載 SD card 分區,並將 `/boot` 中已編譯的檔案複製到 boot partition: ```shell $ sudo mount /dev/sdx1 <path_to_sd_mount_point>/boot $ sudo cp -rf firmware/boot/* <path_to_sd_mount_point>/boot/ $ sync ``` ## 建置 Root File System 1. 下載 ARM64/AArch64 可用的 Linux Distribution Rootfs: ```shell $ wget -c https://rcn-ee.com/rootfs/eewiki/minfs/debian-11.3-minimal-arm64-2022-04-15.tar.xz $ tar Jxvf debian-11.3-minimal-arm64-2022-04-15.tar.xz ``` 2. 將壓縮檔中的 File System 複製到 rootfs partition: ```shell $ sudo tar xf arm64-rootfs-debian-bullseye.tar -C <path_to_sd_mount_point>/rootfs $ sync ``` ## 設定與建置 U-Boot bootloader 1. 於 [Das U-Boot](https://github.com/u-boot) 取得 repository `u-boot` 的資源: ```shell $ git clone --depth=1 https://github.com/u-boot/u-boot ``` 2. 參照 U-Boot 發布的 [Board-specific doc](https://u-boot.readthedocs.io/en/latest/board/broadcom/raspberrypi.html),選擇 Raspberry Pi 4 對應的 `defconfig`,編譯後預期在 `u-boot` 工作目錄中得到二進制檔案 `u-boot.bin`: ```shell $ make CROSS_COMPILE=aarch64-linux-gnu- rpi_arm64_defconfig $ make CROSS_COMPILE=aarch64-linux-gnu- ``` 3. 將 `u-boot.bin` 複製到 boot partition: ```shell $ sudo cp u-boot.bin <path_to_sd_mount_point>/boot $ sync ``` 4. 新增 `boot_cmd.txt`,並加入 boot commands: ```= fatload mmc 0:1 ${kernel_addr_r} Image setenv bootargs 8250.nr_uarts=1 console=ttyS0,115200 root=/dev/mmcblk0p2 rw rootwait loglevel=7 booti ${kernel_addr_r} - ${fdt_addr} ``` 5. 藉由 `mkimage`,得到預期的 boot script,並將 `boot.scr` 複製到 boot partition: ```shell $ mkimage -A arm64 -O linux -T script -C none -d boot_cmd.txt boot.scr $ sudo cp boot.scr <path_to_sd_mount_point>/boot $ sync ``` 6. Raspberry Pi 本身就有預設的 bootloader 能夠來載入 kernel,但實驗預計使用開源的 U-Boot,因此需要對 [Raspberry Pi 系統組態](https://www.raspberrypi.com/documentation/computers/config_txt.html) 進行額外設定,需於 boot partition 中新增 `config.txt`,並加入設定: ```= kernel=u-boot.bin enable_uart=1 arm_64bit=1 ``` ## 設定與建置 Linux kernel 1. 於 [Raspberry Pi](https://github.com/raspberrypi/) 取得 repository `linux` 的資源: ```shell $ git clone --depth=1 https://github.com/raspberrypi/linux ``` 2. 參照 Raspberry Pi 發布的 [Specifications](https://www.raspberrypi.com/products/raspberry-pi-4-model-b/specifications/),選擇 Raspberry Pi 4 對應的 `board_defconfig`,編譯之後預期在 `linux` 工作目錄下得到核心映像檔 `arch/arm64/boot/Image`: ```shell $ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- bcm2711_defconfig $ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- Image modules dtbs ``` 3. 將核心映像檔和 device tree blob (`Image` & `bcm2711-rpi-4-b.dtb`) 複製到 boot partition: ```shell $ sudo cp arch/arm64/boot/Image <path_to_sd_mount_point>/boot $ sudo cp arch/arm64/boot/dts/broadcom/bcm2711-rpi-4-b.dtb <path_to_sd_mount_point>/boot $ sync ``` 4. 安裝核心模組至 rootfs partition: ```shell $ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- INSTALL_MOD_PATH=<path_to_sd_mount_point>/rootfs modules_install ``` ## 建置 Trusted Firmware-A (TF-A) Trusted Firmware-A (TF-A) 是基於 ARMv8-A 以及 ARMv7-A 架構下,對於 [Secure world](https://developer.arm.com/documentation/100935/0100/Interaction-of-Normal-and-Secure-worlds-) software 的參考實作,並且包含 [例外層級](https://developer.arm.com/documentation/102412/0103/Privilege-and-Exception-levels/Exception-levels) 3 (EL3) 的 Secure monitor。 對照 [TrustZone Architecture](https://developer.arm.com/documentation/100935/0100/The-TrustZone-hardware-architecture-), Trusted Firmware-A 已發布的參考實作包含: * EL3 Runtime Firmware * EL3 Boot Frimware * S-EL1 Boot Firmware ![](https://hackmd.io/_uploads/SkAL3Rgd2.jpg) > Reference of Figure 1: [Trusted Firmware Deep Dive](https://www.linaro.org/app/resources/Connect%20Events/Trusted_Firmware_Deep_Dive_v1.0_.pdf) 1. 於 [ARM Software](https://github.com/ARM-software) 取得已發布的 Trusted Firmware Code: ```shell $ git clone -b v2.9 https://github.com/ARM-software/arm-trusted-firmware ``` 依據 Arm 發布的 [Firmware Design Document](https://github.com/ARM-software/arm-trusted-firmware/blob/master/docs/design/firmware-design.rst), TF-A 實作的啟動流程取決於執行狀態,在 AArch64 的架構下可以將啟動流程分為 5 個階段,按照其執行順序分別為: | Boot Loader stage | Execution | | ----------------- | --------------------- | | stage 1 (BL1) | AP Boot ROM | | stage 2 (BL2) | Trusted Boot Firmware | | stage 3-1 (BL31) | EL3 Runtime Firmware | | stage 3-2 (BL32) | Secure-EL1 Payload | | stage 3-3 (BL33) | Non-trusted Firmware | 其中 BL31 的映像檔將由 BL2 進行載入,然後 BL1 會將控制權移交給 BL31 ,並且 BL31 在 AArch64 架構下的特權為 EL3。此外 BL31 會被鏈結並載入到平台特定的 base address。由 BL31 所實作的[功能](https://github.com/ARM-software/arm-trusted-firmware/blob/master/docs/design/firmware-design.rst#aarch64-bl31)包含: * Architectural initialization * Platform initialization * Runtime services initialization * AArch64 BL32 (Secure-EL1 Payload) image initialization * BL33 (Non-trusted Firmware) execution ![](https://hackmd.io/_uploads/Byh4ep-O2.jpg) > Reference of Figure 2: [ARM Trusted Firmware roadmap and progress](https://www.slideshare.net/linaroorg/lcu14-500-arm-trusted-firmware) 2. 參照 [Build Instructions](https://github.com/ARM-software/arm-trusted-firmware/blob/master/docs/plat/rpi4.rst) 以及 [Coding Guidelines](https://github.com/ARM-software/arm-trusted-firmware/blob/master/docs/process/coding-guidelines.rst),啟用並將 log level 設為 `VERBOSE` (we can check logging macro in `debug.h`),編譯後預期在 `arm-trusted-firmware` 工作目錄下得到二進制檔案 `/build/rpi4/debug/bl31.bin`,並將 `bl31.bin` 複製到 boot partition。 ```shell $ CROSS_COMPILE=aarch64-linux-gnu- make PLAT=rpi4 DEBUG=1 LOG_LEVEL=50 $ sudo cp bl31.bin <path_to_sd_mount_point>/boot/ $ sync ``` 3. 為使 Raspberry Pi 4 正確啟動 64-bit kernels,需要在 boot partition 中的 `config.txt` 加入以下設定: ```= enable_uart=1 arm_64bit=1 enable_gic=1 armstub=bl31.bin ``` 4. `umount` 兩個分區 (boot partition & rootfs partition),將 SD card 插回 Raspberry Pi 4 並接上 serial cable 後將其開機,透過 Host 的 [serial console](https://wiki.archlinux.org/title/working_with_the_serial_console#Screen) 來存取和操作 Raspberry Pi 4: ```shell $ sudo screen /dev/ttyUSB0 115200 ``` 5. 藉由 `screen` ,預期看到 Raspberry Pi 4 從開機至啟動核心的 TF-A boot log 以及 kernel log: ```= VERBOSE: rpi4: Preparing to boot 64-bit Linux kernel VERBOSE: Trusted SRAM seen by this BL image: 0x1000 - 0x18000 VERBOSE: Code region: 0x1000 - 0xc000 VERBOSE: Read-only data region: 0xc000 - 0xe000 VERBOSE: Coherent region: 0x17000 - 0x18000 mmap: VA:0x0 PA:0x0 size:0x1000 attr:0x9 granularity:0x40000000 VA:0x1000 PA:0x1000 size:0xb000 attr:0x2 granularity:0x40000000 VA:0xc000 PA:0xc000 size:0x2000 attr:0x42 granularity:0x40000000 VA:0x17000 PA:0x17000 size:0x1000 attr:0x8 granularity:0x40000000 VA:0x1000 PA:0x1000 size:0x17000 attr:0xa granularity:0x40000000 VA:0x2ee00000 PA:0x2ee00000 size:0x400000 attr:0x1a granularity:0x40000000 VA:0xfc000000 PA:0xfc000000 size:0x4000000 attr:0x8 granularity:0x40000000 VERBOSE: Translation tables state: VERBOSE: Xlat regime: EL3 VERBOSE: Max allowed PA: 0xffffffff VERBOSE: Max allowed VA: 0xffffffff VERBOSE: Max mapped PA: 0xffffffff VERBOSE: Max mapped VA: 0xffffffff VERBOSE: Initial lookup level: 1 VERBOSE: Entries @initial lookup level: 4 VERBOSE: Used 3 sub-tables out of 4 (spare: 1) [LV1] VA:0x0 size:0x40000000 [LV2] VA:0x0 size:0x200000 [LV3] VA:0x0 PA:0x0 size:0x1000 NC-RW-XN-S [LV3] VA:0x1000 PA:0x1000 size:0x1000 MEM-RO-EXEC-S [LV3] VA:0x2000 PA:0x2000 size:0x1000 MEM-RO-EXEC-S [LV3] VA:0x3000 PA:0x3000 size:0x1000 MEM-RO-EXEC-S [LV3] VA:0x4000 PA:0x4000 size:0x1000 MEM-RO-EXEC-S (skip...) NOTICE: BL31: v2.9(debug):v2.9.0-180-gd557aaec7 NOTICE: BL31: Built : 21:03:56, Jun 19 2023 INFO: Changed device tree to advertise PSCI. INFO: ARM GICv2 driver initialized INFO: BL31: Initializing runtime services INFO: BL31: cortex_a72: CPU workaround for 859971 was applied WARNING: BL31: cortex_a72: CPU workaround for 1319367 was missing! INFO: BL31: cortex_a72: CPU workaround for cve_2017_5715 was applied INFO: BL31: cortex_a72: CPU workaround for cve_2018_3639 was applied INFO: BL31: cortex_a72: CPU workaround for cve_2022_23960 was applied INFO: BL31: Preparing for EL3 exit to normal world INFO: Entry point address = 0x80000 INFO: SPSR = 0x3c9 VERBOSE: Argument #0 = 0x2eff2700 VERBOSE: Argument #1 = 0x0 VERBOSE: Argument #2 = 0x0 VERBOSE: Argument #3 = 0x0 VERBOSE: Argument #4 = 0x0 VERBOSE: Argument #5 = 0x0 VERBOSE: Argument #6 = 0x0 VERBOSE: Argument #7 = 0x0 U-Boot 2023.07-rc4-g1c30e100 (Jun 19 2023 - 15:09:55 +0800) DRAM: 948 MiB (effective 7.9 GiB) RPI 4 Model B (0xd03115) Core: 210 devices, 16 uclasses, devicetree: board MMC: mmcnr@7e300000: 1, mmc@7e340000: 0 Loading Environment from FAT... Unable to read "uboot.env" from mmc0:1... In: serial Out: serial Err: serial Net: eth0: ethernet@7d580000 PCIe BRCM: link up, 5.0 Gbps x1 (SSC) starting USB... Bus xhci_pci: Register 5000420 NbrPorts 5 Starting the controller USB XHCI 1.00 scanning bus xhci_pci for devices... 4 USB Device(s) found scanning usb for storage devices... 0 Storage Device(s) found Hit any key to stop autoboot: 0 switch to partitions #0, OK mmc0 is current device Scanning mmc 0:1... Found U-Boot script /boot.scr 233 bytes read in 6 ms (37.1 KiB/s) ## Executing script at 02400000 22409728 bytes read in 1887 ms (11.3 MiB/s) Moving Image from 0x80000 to 0x200000, end=1880000 ## Flattened Device Tree blob at 2eff2700 Booting using the fdt blob at 0x2eff2700 Working FDT set to 2eff2700 Using Device Tree in place at 000000002eff2700, end 000000002f00309c Working FDT set to 2eff2700 Starting kernel ... VERBOSE: Unimplemented Standard Service Call: 0x84000050 [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083] [ 0.000000] Linux version 6.1.34-v8+ (root@chien1214-all) (aarch64-linux-gnu-gcc (Ubuntu 11.3.0-1ubuntu1~22.04.1) 11.3.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #1 SMP PREEMPT Mon Jun 19 16:04:08 CST 2023 [ 0.000000] random: crng init done [ 0.000000] Machine model: Raspberry Pi 4 Model B Rev 1.5 (skip...) [ 10.476058] bcmgenet fd580000.ethernet: configuring instance for eternal RGMII (RX delay) [ 10.486477] bcmgenet fd580000.ethernet eth0: Link is Down Starting Network Manager Script Dispatcher Service... [ OK ] Finished Update UTMP about System Runlevel Changes. [ 10.577428] brcmfmac: brcmf_cfg80211_set_power_mgmt: power save enabled [ OK ] Started Network Manager Script Dispatcher Service. [ 11.114793] brcmfmac: brcmf_cfg80211_set_power_mgmt: power save enabled Debian GNU/Linux 11 arm ttyS0 default username:password is [debian:temppwd] arm login: ``` ## TF-A Dynamic Configuration * 參照 [Dynamic Configuration during cold boot](https://github.com/ARM-software/arm-trusted-firmware/blob/master/docs/design/firmware-design.rst#dynamic-configuration-during-cold-boot),每個啟動階段 (BL) 可以按照實作平台的需求進行動態配置。另外 AArch64 的開發平台或開發板提供 trusted ROM 、 trusted SRAM 以及 trusted DRAM (FVP only) 等等的記憶體區域 (MR)。 * 對照先前討論的 5 個啟動階段 (BL1, BL2, BL31, BL32, BL33), TF-A 的參考實作透過暫存器將參數 (e.g. `arg0`) 傳遞到下一個啟動階段,以實際編譯 `/bl31/bl31_main.c` 為例: ```c= void bl31_setup(u_register_t arg0, u_register_t arg1, u_register_t arg2, u_register_t arg3) { /* Perform early platform-specific setup */ bl31_early_platform_setup2(arg0, arg1, arg2, arg3); /* Perform late platform-specific setup */ bl31_plat_arch_setup(); #if CTX_INCLUDE_PAUTH_REGS /* * Assert that the ARMv8.3-PAuth registers are present or an access * fault will be triggered when they are being saved or restored. */ assert(is_armv8_3_pauth_present()); #endif /* CTX_INCLUDE_PAUTH_REGS */ } ``` * 從 `/plat/arm/common/arm_bl2_setup.c` 分析,BL1 透過 `arg1` 將 `meminfo_t` 結構的位址傳遞給 BL2: ```c= void bl2_early_platform_setup2(u_register_t arg0, u_register_t arg1, u_register_t arg2, u_register_t arg3) { arm_bl2_early_platform_setup((uintptr_t)arg0, (meminfo_t *)arg1); generic_delay_timer_init(); } ``` * 從 `/plat/arm/common/arm_bl31_setup.c` (Line 29-30) 分析, BL2 會透過 `arg0` 將要執行的下一個 image 的列表傳遞給 EL3 Runtime software (BL31): ```c= void __init arm_bl31_early_platform_setup(void *from_bl2, uintptr_t soc_fw_config, uintptr_t hw_config, void *plat_params_from_bl2) { /* Initialize the console to provide early debug support */ arm_console_boot_init(); #if RESET_TO_BL31 /* There are no parameters from BL2 if BL31 is a reset vector */ assert(from_bl2 == NULL); assert(plat_params_from_bl2 == NULL); # ifdef BL32_BASE /* Populate entry point information for BL32 */ SET_PARAM_HEAD(&bl32_image_ep_info, PARAM_EP, VERSION_1, 0); SET_SECURITY_STATE(bl32_image_ep_info.h.attr, SECURE); bl32_image_ep_info.pc = BL32_BASE; bl32_image_ep_info.spsr = arm_get_spsr_for_bl32_entry(); #if defined(SPD_spmd) /* SPM (hafnium in secure world) expects SPM Core manifest base address * in x0, which in !RESET_TO_BL31 case loaded after base of non shared * SRAM(after 4KB offset of SRAM). But in RESET_TO_BL31 case all non * shared SRAM is allocated to BL31, so to avoid overwriting of manifest * keep it in the last page. */ bl32_image_ep_info.args.arg0 = ARM_TRUSTED_SRAM_BASE + PLAT_ARM_TRUSTED_SRAM_SIZE - PAGE_SIZE; #endif ``` :::danger :warning: 避免張貼過長的程式碼,而沒有相關的分析和探討 :notes: jserv ::: ## GDB and OpenOCD * 建置 OpenOCD ,參考 ([一](https://github.com/raspberrypi/openocd)) ([二](https://smartobject.gitlab.io/so3/so3_jtag_rpi4.html))。 * Wiring Raspberry Pi 4 with JTAG probe: ![](https://hackmd.io/_uploads/BJ9bANP_h.jpg =75%x) * 參考 [OP-TEE rpi3.rst](https://github.com/OP-TEE/optee_docs/blob/be6f978dcf881f71a8b2420cceba983bcb181b17/building/devices/rpi3.rst#L322),建置 rpi4 實驗用的 Trusted Execution Environment (TEE) * 實驗在 Normal world 和 Secure world 之下, [trusted applications](https://developer.arm.com/tools-and-software/software-development-tools/solutions/secure-applications) 的執行狀況與差異。 > Devices based on Cortex-A processors benefit from TrustZone technology, which provides infrastructure for security in the hardware. These devices feature secure and non-secure processor modes, with a hard boundary between secure and non-secure software, as well as restricted access to hardware resources from non-secure code. ## Boot Flow of TF-A on Raspberry Pi 4 > Each bootloader image can be divided in 2 parts, `PROGBITS` sections and `NOBITS` sections. > For BL31, a platform can specify an alternate location for `NOBITS` sections (other than immediately following PROGBITS sections) by setting `SEPARATE_NOBITS_REGION` to 1 and defining `BL31_NOBITS_BASE` and `BL31_NOBITS_LIMIT`. * 研讀開放教材 [嵌入式系統建構:開發運作於STM32的韌體程式](http://wiki.csie.ncku.edu.tw/embedded/Lab19/stm32-prog.pdf),以釐清 ELF 目的檔與其作用。 * 研讀 [Memory layout of BL images](https://github.com/ARM-software/arm-trusted-firmware/blob/master/docs/design/firmware-design.rst#memory-layout-of-bl-images) ,追蹤編譯的TF-A source code,或透過工具來驗證BL image的載入情形,並參考以下繪製出 rpi4 TF-A Boot Flow:![](https://hackmd.io/_uploads/H1OGO06_2.jpg) > Reference of Figure 3: [LCU14 500 ARM Trusted Firmware](https://www.slideshare.net/linaroorg/lcu14-500-arm-trusted-firmware) * Boot Flow of TF-A on Raspberry Pi 4: