Try   HackMD

Linux 核心專題: Trusted Firmware-A 及 Linux 啟動流程分析

執行人: chien1214

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →
提問清單

  • ?

任務簡述

Trusted Firmware 是 Arm 公司針對 TrustZone 和 Platform Security Architecture 開發的基礎建設，本任務理解 Trusted Firmware-A (TF-A) 及探討 Linux 在 TF-A 的系統啟動流程分析。預計產出:

  • 從 Arm 第一手材料理解 Trusted Firmware 的設計考量和實作規劃，嘗試紀錄和整理
  • 以 Trusted Firmware-A (TF-A) 作為切入，探討 Linux 在 TF-A 的系統啟動流程分析
  • 以 Raspberry Pi 4B 作為主要實驗平台，搭配硬體 JTAG probe 分析和追蹤

TODO: 整理第一手材料

以 Arm Developer 網站為主要依據 (避免羅列 Arm, Linaro, Pengutronix 以外的技術文件，原始程式碼不在此限)，搭配閱讀以下簡報 (和錄影):

解讀針對 Arm 處理器搭配 Trusted Firmware-A (TF-A)，Linux 核心的啟動流程、如何確保 TF-A 的作用，並列舉關鍵程式碼及元件。

TODO: 以 TF-A 切入，探討 Linux 在 TF-A 的系統啟動流程分析

以 Raspberry Pi 4B 作為主要實驗平台，搭配硬體 JTAG probe 分析和追蹤，不僅列出關鍵程式碼，需要搭配實驗來確認關鍵行為。

流程分析

實驗平台與硬體需求

  • Raspberry Pi 4 Model
    • Broadcom BCM2711
    • Quad core Cortex-A72 (ARMv8) 64-bit SoC
    • 8GB LPDDR4-3200 SDRAM
  • Host Linux machine
    • Intel Core i5-4460
    • x86_64
    • Ubuntu 22.04.2 LTS
  • USB to UART TTL Cable 4-pin (PL-2303HX)
  • JTAG debug probe (J-Link)

SD card 磁碟分割與格式化

相較於傳統硬碟 (HDD) 與固態硬碟 (SSD)， Raspberry Pi 使用 SD card 作為儲存裝置，用以存放系統軟體，包含實驗所需的 bootloader、 firmware 以及作業系統。因此 SD card 需要先進行磁碟分割 (2 partitions)，並將其格式化。

  • [Partition #1] - boot partition (FAT)
  • [Partition #2] - rootfs partition (EXT4)
  1. 將 SD card 透過讀卡裝置連接上 Host，並找到系統分配給 SD card 的設備節點 (you may also use lsblk):
$ cat /proc/partitions
major minor  #blocks  name
   
   8       64   30263296 sdx
   8       65     261120 sdx1
   8       66   29998080 sdx2
  1. 使用 fdisk 分割出實驗所需的兩個磁區 (boot partition & rootfs partition):
$ sudo fdisk /dev/sdx
Command (m for help): p

Device     Boot  Start      End  Sectors  Size Id Type
/dev/sdx1  *      8192   530431   522240  255M  c W95 FAT32 (LBA)
/dev/sdx2       530432 60526591 59996160 28.6G 83 Linux
  1. 格式化上述兩個分區:
$ sudo mkfs.vfat -n boot /dev/sdx1
$ sudo mkfs.ext4 -L rootfs /dev/sdx2

Firmware 更新與寫入

  1. 於 rpi firmware 取得 /boot 中的 pre-compiled binaries:
$ git clone --branch stable https://github.com/raspberrypi/firmware
  1. 掛載 SD card 分區，並將 /boot 中已編譯的檔案複製到 boot partition:
$ sudo mount /dev/sdx1 <path_to_sd_mount_point>/boot
$ sudo cp -rf firmware/boot/* <path_to_sd_mount_point>/boot/
$ sync

建置 Root File System

  1. 下載 ARM64/AArch64 可用的 Linux Distribution Rootfs:
$ wget -c https://rcn-ee.com/rootfs/eewiki/minfs/debian-11.3-minimal-arm64-2022-04-15.tar.xz
$ tar Jxvf debian-11.3-minimal-arm64-2022-04-15.tar.xz
  1. 將壓縮檔中的 File System 複製到 rootfs partition:
$ sudo tar xf arm64-rootfs-debian-bullseye.tar -C <path_to_sd_mount_point>/rootfs
$ sync

設定與建置 U-Boot bootloader

  1. 於 Das U-Boot 取得 repository u-boot 的資源:
$ git clone --depth=1 https://github.com/u-boot/u-boot
  1. 參照 U-Boot 發布的 Board-specific doc，選擇 Raspberry Pi 4 對應的 defconfig，編譯後預期在 u-boot 工作目錄中得到二進制檔案 u-boot.bin:
$ make CROSS_COMPILE=aarch64-linux-gnu- rpi_arm64_defconfig
$ make CROSS_COMPILE=aarch64-linux-gnu-
  1. 將 u-boot.bin 複製到 boot partition:
$ sudo cp u-boot.bin <path_to_sd_mount_point>/boot
$ sync
  1. 新增 boot_cmd.txt，並加入 boot commands:



fatload mmc 0:1 ${kernel_addr_r} Image
setenv bootargs 8250.nr_uarts=1 console=ttyS0,115200 root=/dev/mmcblk0p2 rw rootwait loglevel=7
booti ${kernel_addr_r} - ${fdt_addr}
  1. 藉由 mkimage，得到預期的 boot script，並將 boot.scr 複製到 boot partition:
$ mkimage -A arm64 -O linux -T script -C none -d boot_cmd.txt boot.scr
$ sudo cp boot.scr <path_to_sd_mount_point>/boot
$ sync
  1. Raspberry Pi 本身就有預設的 bootloader 能夠來載入 kernel，但實驗預計使用開源的 U-Boot，因此需要對 Raspberry Pi 系統組態 進行額外設定，需於 boot partition 中新增 config.txt，並加入設定:



kernel=u-boot.bin
enable_uart=1
arm_64bit=1

設定與建置 Linux kernel

  1. 於 Raspberry Pi 取得 repository linux 的資源:
$ git clone --depth=1 https://github.com/raspberrypi/linux
  1. 參照 Raspberry Pi 發布的 Specifications，選擇 Raspberry Pi 4 對應的 board_defconfig，編譯之後預期在 linux 工作目錄下得到核心映像檔 arch/arm64/boot/Image:
$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- bcm2711_defconfig
$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- Image modules dtbs
  1. 將核心映像檔和 device tree blob (Image & bcm2711-rpi-4-b.dtb) 複製到 boot partition:
$ sudo cp arch/arm64/boot/Image <path_to_sd_mount_point>/boot
$ sudo cp arch/arm64/boot/dts/broadcom/bcm2711-rpi-4-b.dtb <path_to_sd_mount_point>/boot
$ sync
  1. 安裝核心模組至 rootfs partition:
$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- INSTALL_MOD_PATH=<path_to_sd_mount_point>/rootfs modules_install

建置 Trusted Firmware-A (TF-A)

Trusted Firmware-A (TF-A) 是基於 ARMv8-A 以及 ARMv7-A 架構下，對於 Secure world software 的參考實作，並且包含 例外層級 3 (EL3) 的 Secure monitor。

對照 TrustZone Architecture， Trusted Firmware-A 已發布的參考實作包含:

  • EL3 Runtime Firmware
  • EL3 Boot Frimware
  • S-EL1 Boot Firmware

Reference of Figure 1: Trusted Firmware Deep Dive

  1. 於 ARM Software 取得已發布的 Trusted Firmware Code:
$ git clone -b v2.9 https://github.com/ARM-software/arm-trusted-firmware

依據 Arm 發布的 Firmware Design Document， TF-A 實作的啟動流程取決於執行狀態，在 AArch64 的架構下可以將啟動流程分為 5 個階段，按照其執行順序分別為:

Boot Loader stage Execution
stage 1 (BL1) AP Boot ROM
stage 2 (BL2) Trusted Boot Firmware
stage 3-1 (BL31) EL3 Runtime Firmware
stage 3-2 (BL32) Secure-EL1 Payload
stage 3-3 (BL33) Non-trusted Firmware

其中 BL31 的映像檔將由 BL2 進行載入，然後 BL1 會將控制權移交給 BL31 ，並且 BL31 在 AArch64 架構下的特權為 EL3。此外 BL31 會被鏈結並載入到平台特定的 base address。由 BL31 所實作的功能包含:

  • Architectural initialization
  • Platform initialization
  • Runtime services initialization
  • AArch64 BL32 (Secure-EL1 Payload) image initialization
  • BL33 (Non-trusted Firmware) execution

Reference of Figure 2: ARM Trusted Firmware roadmap and progress

  1. 參照 Build Instructions 以及 Coding Guidelines，啟用並將 log level 設為 VERBOSE (we can check logging macro in debug.h)，編譯後預期在 arm-trusted-firmware 工作目錄下得到二進制檔案 /build/rpi4/debug/bl31.bin，並將 bl31.bin 複製到 boot partition。
$ CROSS_COMPILE=aarch64-linux-gnu- make PLAT=rpi4 DEBUG=1 LOG_LEVEL=50
$ sudo cp bl31.bin <path_to_sd_mount_point>/boot/
$ sync
  1. 為使 Raspberry Pi 4 正確啟動 64-bit kernels，需要在 boot partition 中的 config.txt 加入以下設定:




enable_uart=1
arm_64bit=1
enable_gic=1
armstub=bl31.bin
  1. umount 兩個分區 (boot partition & rootfs partition)，將 SD card 插回 Raspberry Pi 4 並接上 serial cable 後將其開機，透過 Host 的 serial console 來存取和操作 Raspberry Pi 4:
$ sudo screen /dev/ttyUSB0 115200
  1. 藉由 screen ，預期看到 Raspberry Pi 4 從開機至啟動核心的 TF-A boot log 以及 kernel log:














































































































VERBOSE: rpi4: Preparing to boot 64-bit Linux kernel
VERBOSE: Trusted SRAM seen by this BL image: 0x1000 - 0x18000
VERBOSE: Code region: 0x1000 - 0xc000
VERBOSE: Read-only data region: 0xc000 - 0xe000
VERBOSE: Coherent region: 0x17000 - 0x18000
mmap:
 VA:0x0  PA:0x0  size:0x1000  attr:0x9  granularity:0x40000000
 VA:0x1000  PA:0x1000  size:0xb000  attr:0x2  granularity:0x40000000
 VA:0xc000  PA:0xc000  size:0x2000  attr:0x42  granularity:0x40000000
 VA:0x17000  PA:0x17000  size:0x1000  attr:0x8  granularity:0x40000000
 VA:0x1000  PA:0x1000  size:0x17000  attr:0xa  granularity:0x40000000
 VA:0x2ee00000  PA:0x2ee00000  size:0x400000  attr:0x1a  granularity:0x40000000
 VA:0xfc000000  PA:0xfc000000  size:0x4000000  attr:0x8  granularity:0x40000000

VERBOSE: Translation tables state:
VERBOSE:   Xlat regime:     EL3
VERBOSE:   Max allowed PA:  0xffffffff
VERBOSE:   Max allowed VA:  0xffffffff
VERBOSE:   Max mapped PA:   0xffffffff
VERBOSE:   Max mapped VA:   0xffffffff
VERBOSE:   Initial lookup level: 1
VERBOSE:   Entries @initial lookup level: 4
VERBOSE:   Used 3 sub-tables out of 4 (spare: 1)
  [LV1] VA:0x0 size:0x40000000
    [LV2] VA:0x0 size:0x200000
      [LV3] VA:0x0 PA:0x0 size:0x1000 NC-RW-XN-S
      [LV3] VA:0x1000 PA:0x1000 size:0x1000 MEM-RO-EXEC-S
      [LV3] VA:0x2000 PA:0x2000 size:0x1000 MEM-RO-EXEC-S
      [LV3] VA:0x3000 PA:0x3000 size:0x1000 MEM-RO-EXEC-S
      [LV3] VA:0x4000 PA:0x4000 size:0x1000 MEM-RO-EXEC-S
(skip...)

NOTICE:  BL31: v2.9(debug):v2.9.0-180-gd557aaec7
NOTICE:  BL31: Built : 21:03:56, Jun 19 2023
INFO:    Changed device tree to advertise PSCI.
INFO:    ARM GICv2 driver initialized
INFO:    BL31: Initializing runtime services
INFO:    BL31: cortex_a72: CPU workaround for 859971 was applied
WARNING: BL31: cortex_a72: CPU workaround for 1319367 was missing!
INFO:    BL31: cortex_a72: CPU workaround for cve_2017_5715 was applied
INFO:    BL31: cortex_a72: CPU workaround for cve_2018_3639 was applied
INFO:    BL31: cortex_a72: CPU workaround for cve_2022_23960 was applied
INFO:    BL31: Preparing for EL3 exit to normal world
INFO:    Entry point address = 0x80000
INFO:    SPSR = 0x3c9
VERBOSE: Argument #0 = 0x2eff2700
VERBOSE: Argument #1 = 0x0
VERBOSE: Argument #2 = 0x0
VERBOSE: Argument #3 = 0x0
VERBOSE: Argument #4 = 0x0
VERBOSE: Argument #5 = 0x0
VERBOSE: Argument #6 = 0x0
VERBOSE: Argument #7 = 0x0


U-Boot 2023.07-rc4-g1c30e100 (Jun 19 2023 - 15:09:55 +0800)

DRAM:  948 MiB (effective 7.9 GiB)
RPI 4 Model B (0xd03115)
Core:  210 devices, 16 uclasses, devicetree: board
MMC:   mmcnr@7e300000: 1, mmc@7e340000: 0
Loading Environment from FAT... Unable to read "uboot.env" from mmc0:1...
In:    serial
Out:   serial
Err:   serial
Net:   eth0: ethernet@7d580000
PCIe BRCM: link up, 5.0 Gbps x1 (SSC)
starting USB...
Bus xhci_pci: Register 5000420 NbrPorts 5
Starting the controller
USB XHCI 1.00
scanning bus xhci_pci for devices... 4 USB Device(s) found
       scanning usb for storage devices... 0 Storage Device(s) found
Hit any key to stop autoboot:  0
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:1...
Found U-Boot script /boot.scr
233 bytes read in 6 ms (37.1 KiB/s)
## Executing script at 02400000
22409728 bytes read in 1887 ms (11.3 MiB/s)
Moving Image from 0x80000 to 0x200000, end=1880000
## Flattened Device Tree blob at 2eff2700
   Booting using the fdt blob at 0x2eff2700
Working FDT set to 2eff2700
   Using Device Tree in place at 000000002eff2700, end 000000002f00309c
Working FDT set to 2eff2700

Starting kernel ...

VERBOSE: Unimplemented Standard Service Call: 0x84000050
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083]
[    0.000000] Linux version 6.1.34-v8+ (root@chien1214-all) (aarch64-linux-gnu-gcc (Ubuntu 11.3.0-1ubuntu1~22.04.1) 11.3.0,                        GNU ld (GNU Binutils for Ubuntu) 2.38) #1 SMP PREEMPT Mon Jun 19 16:04:08 CST 2023
[    0.000000] random: crng init done
[    0.000000] Machine model: Raspberry Pi 4 Model B Rev 1.5
(skip...)
  
[   10.476058] bcmgenet fd580000.ethernet: configuring instance for eternal RGMII (RX delay)
[   10.486477] bcmgenet fd580000.ethernet eth0: Link is Down
         Starting Network Manager Script Dispatcher Service...
[  OK  ] Finished Update UTMP about System Runlevel Changes.
[   10.577428] brcmfmac: brcmf_cfg80211_set_power_mgmt: power save enabled
[  OK  ] Started Network Manager Script Dispatcher Service.
[   11.114793] brcmfmac: brcmf_cfg80211_set_power_mgmt: power save enabled

Debian GNU/Linux 11 arm ttyS0

default username:password is [debian:temppwd]

arm login:

TF-A Dynamic Configuration

  • 參照 Dynamic Configuration during cold boot，每個啟動階段 (BL) 可以按照實作平台的需求進行動態配置。另外 AArch64 的開發平台或開發板提供 trusted ROM 、 trusted SRAM 以及 trusted DRAM (FVP only) 等等的記憶體區域 (MR)。
  • 對照先前討論的 5 個啟動階段 (BL1, BL2, BL31, BL32, BL33)， TF-A 的參考實作透過暫存器將參數 (e.g. arg0) 傳遞到下一個啟動階段，以實際編譯 /bl31/bl31_main.c 為例:

















void bl31_setup(u_register_t arg0, u_register_t arg1, u_register_t arg2,
                u_register_t arg3)
{
        /* Perform early platform-specific setup */
        bl31_early_platform_setup2(arg0, arg1, arg2, arg3);

        /* Perform late platform-specific setup */
        bl31_plat_arch_setup();

#if CTX_INCLUDE_PAUTH_REGS
        /*
         * Assert that the ARMv8.3-PAuth registers are present or an access
         * fault will be triggered when they are being saved or restored.
         */
        assert(is_armv8_3_pauth_present());
#endif /* CTX_INCLUDE_PAUTH_REGS */
}
  • 從 /plat/arm/common/arm_bl2_setup.c 分析，BL1 透過 arg1 將 meminfo_t 結構的位址傳遞給 BL2:






void bl2_early_platform_setup2(u_register_t arg0, u_register_t arg1, u_register_t arg2, u_register_t arg3)
{
	arm_bl2_early_platform_setup((uintptr_t)arg0, (meminfo_t *)arg1);

	generic_delay_timer_init();
}
  • 從 /plat/arm/common/arm_bl31_setup.c (Line 29-30) 分析， BL2 會透過 arg0 將要執行的下一個 image 的列表傳遞給 EL3 Runtime software (BL31):































void __init arm_bl31_early_platform_setup(void *from_bl2, uintptr_t soc_fw_config,
				uintptr_t hw_config, void *plat_params_from_bl2)
{
	/* Initialize the console to provide early debug support */
	arm_console_boot_init();

#if RESET_TO_BL31
	/* There are no parameters from BL2 if BL31 is a reset vector */
	assert(from_bl2 == NULL);
	assert(plat_params_from_bl2 == NULL);

# ifdef BL32_BASE
	/* Populate entry point information for BL32 */
	SET_PARAM_HEAD(&bl32_image_ep_info,
				PARAM_EP,
				VERSION_1,
				0);
	SET_SECURITY_STATE(bl32_image_ep_info.h.attr, SECURE);
	bl32_image_ep_info.pc = BL32_BASE;
	bl32_image_ep_info.spsr = arm_get_spsr_for_bl32_entry();

#if defined(SPD_spmd)
	/* SPM (hafnium in secure world) expects SPM Core manifest base address
	 * in x0, which in !RESET_TO_BL31 case loaded after base of non shared
	 * SRAM(after 4KB offset of SRAM). But in RESET_TO_BL31 case all non
	 * shared SRAM is allocated to BL31, so to avoid overwriting of manifest
	 * keep it in the last page.
	 */
	bl32_image_ep_info.args.arg0 = ARM_TRUSTED_SRAM_BASE +
				PLAT_ARM_TRUSTED_SRAM_SIZE - PAGE_SIZE;
#endif

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →
避免張貼過長的程式碼，而沒有相關的分析和探討
Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →
jserv

GDB and OpenOCD

  • 建置 OpenOCD ，參考 (一) (二)。
  • Wiring Raspberry Pi 4 with JTAG probe:
  • 參考 OP-TEE rpi3.rst，建置 rpi4 實驗用的 Trusted Execution Environment (TEE)
  • 實驗在 Normal world 和 Secure world 之下， trusted applications 的執行狀況與差異。

Devices based on Cortex-A processors benefit from TrustZone technology, which provides infrastructure for security in the hardware. These devices feature secure and non-secure processor modes, with a hard boundary between secure and non-secure software, as well as restricted access to hardware resources from non-secure code.

Boot Flow of TF-A on Raspberry Pi 4

Each bootloader image can be divided in 2 parts, PROGBITS sections and NOBITS sections.

For BL31, a platform can specify an alternate location for NOBITS sections (other than immediately following PROGBITS sections) by setting SEPARATE_NOBITS_REGION to 1 and defining BL31_NOBITS_BASE and BL31_NOBITS_LIMIT.

Reference of Figure 3: LCU14 500 ARM Trusted Firmware

  • Boot Flow of TF-A on Raspberry Pi 4:
Last changed by 
 
sysprog
System Software Programming
0
2
5218
Published on HackMD