:question: 提問清單
Trusted Firmware 是 Arm 公司針對 TrustZone 和 Platform Security Architecture 開發的基礎建設,本任務理解 Trusted Firmware-A (TF-A) 及探討 Linux 在 TF-A 的系統啟動流程分析。預計產出:
以 Arm Developer 網站為主要依據 (避免羅列 Arm, Linaro, Pengutronix 以外的技術文件,原始程式碼不在此限),搭配閱讀以下簡報 (和錄影):
解讀針對 Arm 處理器搭配 Trusted Firmware-A (TF-A),Linux 核心的啟動流程、如何確保 TF-A 的作用,並列舉關鍵程式碼及元件。
以 Raspberry Pi 4B 作為主要實驗平台,搭配硬體 JTAG probe 分析和追蹤,不僅列出關鍵程式碼,需要搭配實驗來確認關鍵行為。
相較於傳統硬碟 (HDD) 與固態硬碟 (SSD), Raspberry Pi 使用 SD card 作為儲存裝置,用以存放系統軟體,包含實驗所需的 bootloader、 firmware 以及作業系統。因此 SD card 需要先進行磁碟分割 (2 partitions),並將其格式化。
lsblk
):$ cat /proc/partitions
major minor #blocks name
8 64 30263296 sdx
8 65 261120 sdx1
8 66 29998080 sdx2
fdisk
分割出實驗所需的兩個磁區 (boot partition & rootfs partition):$ sudo fdisk /dev/sdx
Command (m for help): p
Device Boot Start End Sectors Size Id Type
/dev/sdx1 * 8192 530431 522240 255M c W95 FAT32 (LBA)
/dev/sdx2 530432 60526591 59996160 28.6G 83 Linux
$ sudo mkfs.vfat -n boot /dev/sdx1
$ sudo mkfs.ext4 -L rootfs /dev/sdx2
/boot
中的 pre-compiled binaries:$ git clone --branch stable https://github.com/raspberrypi/firmware
/boot
中已編譯的檔案複製到 boot partition:$ sudo mount /dev/sdx1 <path_to_sd_mount_point>/boot
$ sudo cp -rf firmware/boot/* <path_to_sd_mount_point>/boot/
$ sync
$ wget -c https://rcn-ee.com/rootfs/eewiki/minfs/debian-11.3-minimal-arm64-2022-04-15.tar.xz
$ tar Jxvf debian-11.3-minimal-arm64-2022-04-15.tar.xz
$ sudo tar xf arm64-rootfs-debian-bullseye.tar -C <path_to_sd_mount_point>/rootfs
$ sync
u-boot
的資源:$ git clone --depth=1 https://github.com/u-boot/u-boot
defconfig
,編譯後預期在 u-boot
工作目錄中得到二進制檔案 u-boot.bin
:$ make CROSS_COMPILE=aarch64-linux-gnu- rpi_arm64_defconfig
$ make CROSS_COMPILE=aarch64-linux-gnu-
u-boot.bin
複製到 boot partition:$ sudo cp u-boot.bin <path_to_sd_mount_point>/boot
$ sync
boot_cmd.txt
,並加入 boot commands:
fatload mmc 0:1 ${kernel_addr_r} Image
setenv bootargs 8250.nr_uarts=1 console=ttyS0,115200 root=/dev/mmcblk0p2 rw rootwait loglevel=7
booti ${kernel_addr_r} - ${fdt_addr}
mkimage
,得到預期的 boot script,並將 boot.scr
複製到 boot partition:$ mkimage -A arm64 -O linux -T script -C none -d boot_cmd.txt boot.scr
$ sudo cp boot.scr <path_to_sd_mount_point>/boot
$ sync
config.txt
,並加入設定:
kernel=u-boot.bin
enable_uart=1
arm_64bit=1
linux
的資源:$ git clone --depth=1 https://github.com/raspberrypi/linux
board_defconfig
,編譯之後預期在 linux
工作目錄下得到核心映像檔 arch/arm64/boot/Image
:$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- bcm2711_defconfig
$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- Image modules dtbs
Image
& bcm2711-rpi-4-b.dtb
) 複製到 boot partition:$ sudo cp arch/arm64/boot/Image <path_to_sd_mount_point>/boot
$ sudo cp arch/arm64/boot/dts/broadcom/bcm2711-rpi-4-b.dtb <path_to_sd_mount_point>/boot
$ sync
$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- INSTALL_MOD_PATH=<path_to_sd_mount_point>/rootfs modules_install
Trusted Firmware-A (TF-A) 是基於 ARMv8-A 以及 ARMv7-A 架構下,對於 Secure world software 的參考實作,並且包含 例外層級 3 (EL3) 的 Secure monitor。
對照 TrustZone Architecture, Trusted Firmware-A 已發布的參考實作包含:
Reference of Figure 1: Trusted Firmware Deep Dive
$ git clone -b v2.9 https://github.com/ARM-software/arm-trusted-firmware
依據 Arm 發布的 Firmware Design Document, TF-A 實作的啟動流程取決於執行狀態,在 AArch64 的架構下可以將啟動流程分為 5 個階段,按照其執行順序分別為:
Boot Loader stage | Execution |
---|---|
stage 1 (BL1) | AP Boot ROM |
stage 2 (BL2) | Trusted Boot Firmware |
stage 3-1 (BL31) | EL3 Runtime Firmware |
stage 3-2 (BL32) | Secure-EL1 Payload |
stage 3-3 (BL33) | Non-trusted Firmware |
其中 BL31 的映像檔將由 BL2 進行載入,然後 BL1 會將控制權移交給 BL31 ,並且 BL31 在 AArch64 架構下的特權為 EL3。此外 BL31 會被鏈結並載入到平台特定的 base address。由 BL31 所實作的功能包含:
Reference of Figure 2: ARM Trusted Firmware roadmap and progress
VERBOSE
(we can check logging macro in debug.h
),編譯後預期在 arm-trusted-firmware
工作目錄下得到二進制檔案 /build/rpi4/debug/bl31.bin
,並將 bl31.bin
複製到 boot partition。$ CROSS_COMPILE=aarch64-linux-gnu- make PLAT=rpi4 DEBUG=1 LOG_LEVEL=50
$ sudo cp bl31.bin <path_to_sd_mount_point>/boot/
$ sync
config.txt
加入以下設定:
enable_uart=1
arm_64bit=1
enable_gic=1
armstub=bl31.bin
umount
兩個分區 (boot partition & rootfs partition),將 SD card 插回 Raspberry Pi 4 並接上 serial cable 後將其開機,透過 Host 的 serial console 來存取和操作 Raspberry Pi 4:$ sudo screen /dev/ttyUSB0 115200
screen
,預期看到 Raspberry Pi 4 從開機至啟動核心的 TF-A boot log 以及 kernel log:
VERBOSE: rpi4: Preparing to boot 64-bit Linux kernel
VERBOSE: Trusted SRAM seen by this BL image: 0x1000 - 0x18000
VERBOSE: Code region: 0x1000 - 0xc000
VERBOSE: Read-only data region: 0xc000 - 0xe000
VERBOSE: Coherent region: 0x17000 - 0x18000
mmap:
VA:0x0 PA:0x0 size:0x1000 attr:0x9 granularity:0x40000000
VA:0x1000 PA:0x1000 size:0xb000 attr:0x2 granularity:0x40000000
VA:0xc000 PA:0xc000 size:0x2000 attr:0x42 granularity:0x40000000
VA:0x17000 PA:0x17000 size:0x1000 attr:0x8 granularity:0x40000000
VA:0x1000 PA:0x1000 size:0x17000 attr:0xa granularity:0x40000000
VA:0x2ee00000 PA:0x2ee00000 size:0x400000 attr:0x1a granularity:0x40000000
VA:0xfc000000 PA:0xfc000000 size:0x4000000 attr:0x8 granularity:0x40000000
VERBOSE: Translation tables state:
VERBOSE: Xlat regime: EL3
VERBOSE: Max allowed PA: 0xffffffff
VERBOSE: Max allowed VA: 0xffffffff
VERBOSE: Max mapped PA: 0xffffffff
VERBOSE: Max mapped VA: 0xffffffff
VERBOSE: Initial lookup level: 1
VERBOSE: Entries @initial lookup level: 4
VERBOSE: Used 3 sub-tables out of 4 (spare: 1)
[LV1] VA:0x0 size:0x40000000
[LV2] VA:0x0 size:0x200000
[LV3] VA:0x0 PA:0x0 size:0x1000 NC-RW-XN-S
[LV3] VA:0x1000 PA:0x1000 size:0x1000 MEM-RO-EXEC-S
[LV3] VA:0x2000 PA:0x2000 size:0x1000 MEM-RO-EXEC-S
[LV3] VA:0x3000 PA:0x3000 size:0x1000 MEM-RO-EXEC-S
[LV3] VA:0x4000 PA:0x4000 size:0x1000 MEM-RO-EXEC-S
(skip...)
NOTICE: BL31: v2.9(debug):v2.9.0-180-gd557aaec7
NOTICE: BL31: Built : 21:03:56, Jun 19 2023
INFO: Changed device tree to advertise PSCI.
INFO: ARM GICv2 driver initialized
INFO: BL31: Initializing runtime services
INFO: BL31: cortex_a72: CPU workaround for 859971 was applied
WARNING: BL31: cortex_a72: CPU workaround for 1319367 was missing!
INFO: BL31: cortex_a72: CPU workaround for cve_2017_5715 was applied
INFO: BL31: cortex_a72: CPU workaround for cve_2018_3639 was applied
INFO: BL31: cortex_a72: CPU workaround for cve_2022_23960 was applied
INFO: BL31: Preparing for EL3 exit to normal world
INFO: Entry point address = 0x80000
INFO: SPSR = 0x3c9
VERBOSE: Argument #0 = 0x2eff2700
VERBOSE: Argument #1 = 0x0
VERBOSE: Argument #2 = 0x0
VERBOSE: Argument #3 = 0x0
VERBOSE: Argument #4 = 0x0
VERBOSE: Argument #5 = 0x0
VERBOSE: Argument #6 = 0x0
VERBOSE: Argument #7 = 0x0
U-Boot 2023.07-rc4-g1c30e100 (Jun 19 2023 - 15:09:55 +0800)
DRAM: 948 MiB (effective 7.9 GiB)
RPI 4 Model B (0xd03115)
Core: 210 devices, 16 uclasses, devicetree: board
MMC: mmcnr@7e300000: 1, mmc@7e340000: 0
Loading Environment from FAT... Unable to read "uboot.env" from mmc0:1...
In: serial
Out: serial
Err: serial
Net: eth0: ethernet@7d580000
PCIe BRCM: link up, 5.0 Gbps x1 (SSC)
starting USB...
Bus xhci_pci: Register 5000420 NbrPorts 5
Starting the controller
USB XHCI 1.00
scanning bus xhci_pci for devices... 4 USB Device(s) found
scanning usb for storage devices... 0 Storage Device(s) found
Hit any key to stop autoboot: 0
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:1...
Found U-Boot script /boot.scr
233 bytes read in 6 ms (37.1 KiB/s)
## Executing script at 02400000
22409728 bytes read in 1887 ms (11.3 MiB/s)
Moving Image from 0x80000 to 0x200000, end=1880000
## Flattened Device Tree blob at 2eff2700
Booting using the fdt blob at 0x2eff2700
Working FDT set to 2eff2700
Using Device Tree in place at 000000002eff2700, end 000000002f00309c
Working FDT set to 2eff2700
Starting kernel ...
VERBOSE: Unimplemented Standard Service Call: 0x84000050
[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083]
[ 0.000000] Linux version 6.1.34-v8+ (root@chien1214-all) (aarch64-linux-gnu-gcc (Ubuntu 11.3.0-1ubuntu1~22.04.1) 11.3.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #1 SMP PREEMPT Mon Jun 19 16:04:08 CST 2023
[ 0.000000] random: crng init done
[ 0.000000] Machine model: Raspberry Pi 4 Model B Rev 1.5
(skip...)
[ 10.476058] bcmgenet fd580000.ethernet: configuring instance for eternal RGMII (RX delay)
[ 10.486477] bcmgenet fd580000.ethernet eth0: Link is Down
Starting Network Manager Script Dispatcher Service...
[ OK ] Finished Update UTMP about System Runlevel Changes.
[ 10.577428] brcmfmac: brcmf_cfg80211_set_power_mgmt: power save enabled
[ OK ] Started Network Manager Script Dispatcher Service.
[ 11.114793] brcmfmac: brcmf_cfg80211_set_power_mgmt: power save enabled
Debian GNU/Linux 11 arm ttyS0
default username:password is [debian:temppwd]
arm login:
arg0
) 傳遞到下一個啟動階段,以實際編譯 /bl31/bl31_main.c
為例:
void bl31_setup(u_register_t arg0, u_register_t arg1, u_register_t arg2,
u_register_t arg3)
{
/* Perform early platform-specific setup */
bl31_early_platform_setup2(arg0, arg1, arg2, arg3);
/* Perform late platform-specific setup */
bl31_plat_arch_setup();
#if CTX_INCLUDE_PAUTH_REGS
/*
* Assert that the ARMv8.3-PAuth registers are present or an access
* fault will be triggered when they are being saved or restored.
*/
assert(is_armv8_3_pauth_present());
#endif /* CTX_INCLUDE_PAUTH_REGS */
}
/plat/arm/common/arm_bl2_setup.c
分析,BL1 透過 arg1
將 meminfo_t
結構的位址傳遞給 BL2:
void bl2_early_platform_setup2(u_register_t arg0, u_register_t arg1, u_register_t arg2, u_register_t arg3)
{
arm_bl2_early_platform_setup((uintptr_t)arg0, (meminfo_t *)arg1);
generic_delay_timer_init();
}
/plat/arm/common/arm_bl31_setup.c
(Line 29-30) 分析, BL2 會透過 arg0
將要執行的下一個 image 的列表傳遞給 EL3 Runtime software (BL31):
void __init arm_bl31_early_platform_setup(void *from_bl2, uintptr_t soc_fw_config,
uintptr_t hw_config, void *plat_params_from_bl2)
{
/* Initialize the console to provide early debug support */
arm_console_boot_init();
#if RESET_TO_BL31
/* There are no parameters from BL2 if BL31 is a reset vector */
assert(from_bl2 == NULL);
assert(plat_params_from_bl2 == NULL);
# ifdef BL32_BASE
/* Populate entry point information for BL32 */
SET_PARAM_HEAD(&bl32_image_ep_info,
PARAM_EP,
VERSION_1,
0);
SET_SECURITY_STATE(bl32_image_ep_info.h.attr, SECURE);
bl32_image_ep_info.pc = BL32_BASE;
bl32_image_ep_info.spsr = arm_get_spsr_for_bl32_entry();
#if defined(SPD_spmd)
/* SPM (hafnium in secure world) expects SPM Core manifest base address
* in x0, which in !RESET_TO_BL31 case loaded after base of non shared
* SRAM(after 4KB offset of SRAM). But in RESET_TO_BL31 case all non
* shared SRAM is allocated to BL31, so to avoid overwriting of manifest
* keep it in the last page.
*/
bl32_image_ep_info.args.arg0 = ARM_TRUSTED_SRAM_BASE +
PLAT_ARM_TRUSTED_SRAM_SIZE - PAGE_SIZE;
#endif
:warning: 避免張貼過長的程式碼,而沒有相關的分析和探討
:notes: jserv
Devices based on Cortex-A processors benefit from TrustZone technology, which provides infrastructure for security in the hardware. These devices feature secure and non-secure processor modes, with a hard boundary between secure and non-secure software, as well as restricted access to hardware resources from non-secure code.
Each bootloader image can be divided in 2 parts,
PROGBITS
sections andNOBITS
sections.
For BL31, a platform can specify an alternate location for
NOBITS
sections (other than immediately following PROGBITS sections) by settingSEPARATE_NOBITS_REGION
to 1 and definingBL31_NOBITS_BASE
andBL31_NOBITS_LIMIT
.
Reference of Figure 3: LCU14 500 ARM Trusted Firmware