Load Firmware Files Later in Linux Kernel

Some Linux device drivers need loading firmware files to enable the features, especially the new DRM and Wireless drivers.

However, we notice some drivers load the firmware files when they probe, not the open action. If the driver is built in kernel image and it will load a firmware file, then it may fail to load the firmware file. Because, the kernel may have not mounted the whole file system before probing, yet. The current workaround is building the drivers as modules and inserting the firmware files into the ramfs. The WiFi on Raspberry Pi has the same issue as well.

So, the initial idea is that try to move the firmware loading action from probe to open function for Raspberry Pi's WiFi driver. But, it is good to study how other drivers do related things in advance.

How do other WiFi modules load firmware files

Let's check the Intel WiFi on my laptop as the example!

$ sudo dmesg | grep "iwlwifi"
[    4.565153] iwlwifi 0000:00:14.3: enabling device (0000 -> 0002)
[    4.567869] iwlwifi 0000:00:14.3: Direct firmware load for iwlwifi-so-a0-gf-a0-72.ucode failed with error -2
[    4.611449] iwlwifi 0000:00:14.3: api flags index 2 larger than supported by driver
[    4.611464] iwlwifi 0000:00:14.3: TLV_FW_FSEQ_VERSION: FSEQ Version: 0.0.2.36
[    4.611752] iwlwifi 0000:00:14.3: loaded firmware version 71.058653f6.0 so-a0-gf-a0-71.ucode op_mode iwlmvm
[    4.866730] iwlwifi 0000:00:14.3: Detected Intel(R) Wi-Fi 6E AX211 160MHz, REV=0x370
[    5.043036] iwlwifi 0000:00:14.3: loaded PNVM version 881c99e1
[    5.058124] iwlwifi 0000:00:14.3: Detected RF GF, rfid=0x2010d000
[    5.127537] iwlwifi 0000:00:14.3: base HW address: 74:04:f1:47:d9:39
[    9.381929] iwlwifi 0000:00:14.3 wlp0s20f3: renamed from wlan0

The module iwlwifi loads "71.058653f6.0 so-a0-gf-a0-71.ucode" in the filesystem.

$ git grep "loaded firmware version"
drivers/net/ethernet/cavium/liquidio/lio_main.c:                                 "Using auto-loaded firmware version %s.\n",
drivers/net/wireless/intel/iwlegacy/3945-mac.c: IL_INFO("loaded firmware version %u.%u.%u.%u\n",
drivers/net/wireless/intel/iwlegacy/4965-mac.c: IL_INFO("loaded firmware version %u.%u.%u.%u\n",
drivers/net/wireless/intel/iwlwifi/iwl-drv.c:   IWL_INFO(drv, "loaded firmware version %s op_mode %s\n",
drivers/staging/media/ipu3/ipu3-css-fw.c:       dev_info(dev, "loaded firmware version %.64s, %u binaries, %zu bytes\n",

So, it is in drivers/net/wireless/intel/iwlwifi/iwl-drv.c's iwl_req_fw_callback().

Intel WiFi has a Driver system flows:

/**
 * DOC: Driver system flows - drv component
 *
 * This component implements the system flows such as bus enumeration, bus
 * removal. Bus dependent parts of system flows (such as iwl_pci_probe) are in
 * bus specific files (transport files). This is the code that is common among
 * different buses.
 *
 * This component is also in charge of managing the several implementations of
 * the wifi flows: it will allow to have several fw API implementation. These
 * different implementations will differ in the way they implement mac80211's
 * handlers too.

 * The init flow wrt to the drv component looks like this:
 * 1) The bus specific component is called from module_init
 * 2) The bus specific component registers the bus driver
 * 3) The bus driver calls the probe function
 * 4) The bus specific component configures the bus
 * 5) The bus specific component calls to the drv bus agnostic part
 *    (iwl_drv_start)
 * 6) iwl_drv_start fetches the fw ASYNC, iwl_req_fw_callback
 * 7) iwl_req_fw_callback parses the fw file
 * 8) iwl_req_fw_callback starts the wifi implementation to matches the fw
 */

Trace the real code:
module_init(iwl_drv_init) -> iwl_pci_register_driver() -> iwl_pci_driver -> iwl_pci_probe() -> iwl_drv_start() -> iwl_request_firmware() -> iwl_req_fw_callback()

The driver is registered as a kind of PCI driver:

static struct pci_driver iwl_pci_driver = {
	.name = DRV_NAME,
	.id_table = iwl_hw_card_ids,
	.probe = iwl_pci_probe,
	.remove = iwl_pci_remove,
	.driver.pm = IWL_PM_OPS,
};

int __must_check iwl_pci_register_driver(void)
{
	int ret;
	ret = pci_register_driver(&iwl_pci_driver);
	if (ret)
		pr_err("Unable to initialize PCI module\n");

	return ret;
}

The pci_driver has probe, remove … functions, but does not have the open function.

Then, moving the load firmware file action into open function might be impractical. :cry:

How does a device driver load the firmware

Following the Intel WiFi driver, it loads the firmware files by the procedure:
request_firmware_nowait(THIS_MODULE, 1, drv->firmware_name, drv->trans->dev, GFP_KERNEL, drv, iwl_req_fw_callback) -> request_firmware_work_func() -> _request_firmware() -> fw_get_filesystem_firmware()

I notice fw_get_filesystem_firmware() is blocked by wait_for_initramfs() introduced from commit e7cb072eb988 ("init/initramfs.c: do unpacking asynchronously").

This means if any driver wants to load the firmware file from filesystem, it waits until the rootfs in the ramfs populated. Interesting!

How about wait until root filesystem is mounted

Trace the root partition mounting

The root device (root=) is set in root_dev_setup().

From Early userspace support - How does it work?, the kernel has currently 3 ways to mount the root filesystem:

• all required device and filesystem drivers compiled into the kernel, no initrd. init/main.c:init() will call prepare_namespace() to mount the final root filesystem, based on the root= option and optional init= to run some other init binary than listed at the end of init/main.c:init().
• some device and filesystem drivers built as modules and stored in an initrd. The initrd must contain a binary ‘/linuxrc’ which is supposed to load these driver modules. It is also possible to mount the final root filesystem via linuxrc and use the pivot_root syscall. The initrd is mounted and executed via prepare_namespace().
• using initramfs. The call to prepare_namespace() must be skipped. This means that a binary must do all the work. Said binary can be stored into initramfs either via modifying usr/gen_init_cpio.c or via the new initrd format, an cpio archive. It must be called “/init”. This binary is responsible to do all the things prepare_namespace() would do.

To maintain backwards compatibility, the /init binary will only run if it comes via an initramfs cpio archive. If this is not the case, init/main.c:init() will run prepare_namespace() to mount the final root and exec one of the predefined init binaries.

prepare_namespace() is an entry point to mount the root filesystem. However, …

Distro with initramfs

Linux distros like Endless OS use the initramfs. Then, the call to prepare_namespace() must be skipped. The real root filesystem is mounted by OSTree, not the code in the kernel.

So, checking root file system is mounted cannot be the barrier in the device driver.

The interface of Intel WiFi

Thanks to Daniel's reminder:
The Intel WiFi driver is not only a PCI device driver, but also a WiFi interface driver. The WiFi feature cannot be access without the interface.

From this angle, found the WiFi interface driver has the start() callback function.

The interface of RPi 4B's WiFi

This is the test target.

• The driver is brcmfmac: Broadcom 802.11 wireless LAN fullmac driver.
• Because it is fullmac driver, the interface should be cfg80211_ops.