Using Rust for a Linux driver

# Using Rust for a Linux driver ## Rust and C glue ### Weird unwind requirements After hacking a small rust module together, `modpost` decided to hit us with the following error: ```sh ERROR: modpost: "__aeabi_unwind_cpp_pr1" [/home/kagounard/Git/ks0n/mfrc522-linux/rust-module/mfrc522.ko] undefined! ERROR: modpost: "__aeabi_unwind_cpp_pr0" [/home/kagounard/Git/ks0n/mfrc522-linux/rust-module/mfrc522.ko] undefined! ``` Despite us not using anything unwind related (`panic = abort`, `#[unwind(aborts)]`, etc...). This was very hard to try and solve, as not many people are using rustc with modpost yet. Furthermore, this was without the use of Rust-for-Linux, so we didn't have anyone to ask. It seems that this error is quite present in the Android dev community, as they often cross-compile for ARM devices. Some online searches lead toward a lack of linking with `libgcc`, and despite our best efforts towards that goal, we didn't manage to fix the missing symbols. An alternative was then to define them ourselves! We wrote some glue code in order to shut modpost up and link properly: ```rust= // If we arrive here, we're screwed anyway #[no_mangle] unsafe extern "C" fn __aeabi_unwind_cpp_pr1() -> ! { panic!() } #[no_mangle] unsafe extern "C" fn __aeabi_unwind_cpp_pr0() -> ! { panic!() } ``` Sadly, the kernel complained about missing relocations upon `insmod` of our module. Funnily enough, this bug was not present when not using "advanced" Rust constructs: Pattern matching, traits... But what is the point of Rust if you cannot use anything Rust provides? ### Making our glue One more disadvantage of the "manual" version of a Rust driver was that we had to provide the Kernel glue ourselves. We couldn't get `bindgen` (more on that later...) to work properly, and the prospect of writing it by hand was not a happy one. ## `linux-next` We were therefore not huge fans of the Rust and C glue, and all the workarounds it required. During our development, a patch enabling Rust kernel module development was merged into linux-next. We chose to drop the glue and to try developing the module this way. ### Getting a kernel with Rust to compile ```sh BINDGEN rust/bindings_generated.rs EXPORTS rust/exports_core_generated.h RUSTC L rust/compiler_builtins.o llvm-objcopy: '__*': No such file or directory make[1]: *** [rust/Makefile:140: rust/compiler_builtins.o] Error 1 make[1]: *** Deleting file 'rust/compiler_builtins.o' make[1]: *** Waiting for unfinished jobs.... make: *** [Makefile:1314: prepare0] Error 2 ``` This error was due to an out of date version of LLVM. To build linux-next with Rust support, you need to have LLVM >= 10.0.1, which we did not have on our compilation server. We made the switch, but we still spent quite some time trying to tweak different kernel makefiles so that `llvm-objcopy` would get correct arguments. ### Unknown rust target ```sh error[E0463]: can't find crate for `core` │································· | │································· = note: the `target-10773475644222393364` target may not be installed │································· │································· error: aborting due to previous error │································· │································· For more information about this error, try `rustc --explain E0463`. ``` This error popped when trying to cross compile the kernel, and later on when trying to compile an out-of-tree module. This is seemingly unsolvable, and absolutely zero resource regarding this exist online. Even when using `rustup` to install the `aarch64-unknown-linux-gnu` target and following the [Quickstart guide](https://github.com/Rust-for-Linux/linux/blob/rust/Documentation/rust/quick-start.rst), cross-compilation eluded us. The alternative was to use an aarch64 `chroot`. ### Chrooting our way up #### OS choice Since Ubuntu did not offer any version of clang/llvm more recent than 10.0.1 at the time, we chose to go with [ArchLinux](http://os.archlinuxarm.org/os/ArchLinuxARM-aarch64-latest.tar.gz). This is an `aarch64` rootfs, so it couldn't work on our x86_64 machine... unless we were using [``qemu-user-binfmt``](https://packages.ubuntu.com/focal/qemu-user-binfmt). Since this basically emulates an `aarch64` CPU, single core performence is bysmal (but still way better than the RaspberryPi itself). Our saving grace was to use an 80 cores machine, which made compilation faster. The incremental build times were still important, sitting around 5 to 10 minutes. #### DNS not working In order to download the clang toolchain and rust toolchain we needed to download and install the packages. But inside the chroot, the DNS seemed to not work correctly. Here is the fix that we found: ```shell= # /mnt is where the chroot is mount /run /mnt/run chroot /mnt echo 'nameserver 8.8.8.8' > /etc/resolv.conf ``` This seems to be Ubuntu specific. After doing this, you might have DNS issues on the host which a reboot will fix. #### Missing /etc/mtab In order to get a working ArchLinux chroot, you need to use the [``arch-chroot``](https://git.archlinux.org/arch-install-scripts.git) utility. #### Compiling linux-next While the last few problems seemed very straightforward, they accumulated and made us lose a lot of time, which we already had little of. From now on, we stopped doing any weird-stuff, and started developing in a more straight forward way. Since we had to modify the [Rust-for-linux](https://github.com/Rust-for-Linux/linux) fork of the kernel, we could work on our machines and compile the kernel for x86. Thankfully, rust offers excellent compatibility accross targets and easy cross-compilation, so we knew that what was compiling on our development machines would compile on `aarch64`. ### Adding the spi interface to Rust `bindgen` is a tool used to create a bridge between existing C code and newly created Rust code. It creates (relatively) safe bindings for every structure, function and type in the provided headers. However, the generated functions are still external to the Rust code, and therefore unsafe to call, which means that Rust cannot guarantee that nothing shady is happening behind the scenes. Since this unsafe code is scary to call, as well as annoying to use, we need to create high level abstractions allowing kernel developers to use safe Rust code. You can find the existing abstractions [here](https://github.com/Rust-for-Linux/linux/tree/rust/rust/kernel). We set out to create an `Spi` abstraction, allowing users to call into the SPI driver of the kernel: `spi_driver`, `spi_device`, `spi_write_then_read()`... The default configuration for `bindgen` did not generate any SPI rust bindings. We forked the [Rust-for-Linux/linux](https://github.com/Rust-for-Linux/linux.git) repository and started developing. You can find the fork at the [following address](https://github.com/ks0n/linux). The first step was understanding how to create the bindings, which are generated from a set of headers located [here](https://github.com/Rust-for-Linux/linux/blob/rust/rust/kernel/bindings_helper.h). We added the spi header, hid it behind a `#ifdef CONFIG_SPI` to not burden non-embedded users of linux, and got our bindings. __Side-note__: Using the *wrong* version of bindgen (not 0.56.0) will cause inappropriate bindings to be generated. ### No internet Upon flashing our newly compiled kernel onto the Raspberry Pi, we lost access to the ethernet and wifi card. Since our workflow relied on `ssh`-ing into the device to copy and load modules, we got stopped. After a lot of tweaking of config files and a lot of research, we finally found and enabled the wifi card driver module. We still had no ethernet though. ### NetworkManager/wpa_supplicant stop job For some reason we couldn't reboot properly. `NetworkManager` and `wpa_supplicant` were not killed and our Raspberry Pi hung without rebooting... So we needed to have it close by in order to physically power off the board, which meant we could not work on the driver from home or after the curfew. (We later discovered the `-f` flag of the `reboot` binary. Who has time to read manpages anyway?) ### Module Crash upon seemingly unrelated call to dev_uevent/ksys_read? After SPI Probe? ```d [ 36.266704] mfrc522: [MFRC522-RS] Init [ 36.267087] mfrc522: [MFRC522-RS] Get probed, nerd [ 36.272868] Unable to handle kernel paging request at virtual address ffff80001087b7d0 [ 36.282778] Mem abort info: [ 36.286541] ESR = 0x96000007 [ 36.290281] EC = 0x25: DABT (current EL), IL = 32 bits [ 36.296322] SET = 0, FnV = 0 [ 36.300027] EA = 0, S1PTW = 0 [ 36.303836] Data abort info: [ 36.307375] ISV = 0, ISS = 0x00000007 [ 36.311887] CM = 0, WnR = 0 [ 36.315514] swapper pgtable: 4k pages, 48-bit VAs, pgdp=00000000013f1000 [ 36.322967] [ffff80001087b7d0] pgd=0000000001810003, p4d=0000000001810003, pud=0000000001811003, pmd=00000000039b9003, pte=0000000000000000 [ 36.337034] Internal error: Oops: 96000007 [#1] PREEMPT SMP [ 36.343378] Modules linked in: mfrc522 hci_uart btqca btbcm bluetooth ecdh_generic ecc 8021q mrp garp stp llc spidev brcmfmac vc4 cec brcmutil drm_kms_helper cfg80211 drm raspberrypi_cpufreq rfkill clk_raspberrypi raspberrypi_hwmon crct10dif_ce i2c_bcm2835 bcm2835_thermal bcm2835_rng rng_core spi_bcm2835 aes_neon_bs aes_neon_blk ip_tables x_tables ipv6 [ 36.378161] CPU: 3 PID: 539 Comm: systemd-udevd Tainted: G W 5.12.0-rc4+ #25 [ 36.388189] Hardware name: Raspberry Pi 3 Model B Plus Rev 1.3 (DT) [ 36.395343] pstate: 80000005 (Nzcv daif -PAN -UAO -TCO BTYPE=--) [ 36.402250] pc : dev_uevent+0x138/0x1e8 [ 36.406966] lr : uevent_show+0x90/0x118 [ 36.411677] sp : ffff80001079bba0 [ 36.415849] x29: ffff80001079bbc0 x28: ffff670b41b9d400 [ 36.422066] x27: 0000000000000000 x26: 0000000000000001 [ 36.428281] x25: 0000000000000000 x24: ffff670b49b42700 [ 36.434496] x23: ffffdc99f37a4168 x22: ffff670b4191a880 [ 36.440711] x21: ffff670b42475800 x20: ffff670b47f59000 [ 36.446900] x19: ffff670b42475800 x18: 0000000000000000 [ 36.453060] x17: 0000000000000000 x16: 0000000000000000 [ 36.459216] x15: ffff670b4342f0d0 x14: 0000000000000000 [ 36.465374] x13: 0000000000000001 x12: 0000000000000000 [ 36.471522] x11: 0000000000000001 x10: ffff670b47f58000 [ 36.477660] x9 : 0de4e88efe7dd600 x8 : ffff80001087b7d0 [ 36.483806] x7 : 0000000000000000 x6 : 000000000000003f [ 36.489957] x5 : 0000000000000040 x4 : ffff80001079bb80 [ 36.496107] x3 : 0000000000000001 x2 : ffff670b47f59000 [ 36.502254] x1 : ffff670b42475800 x0 : ffff670b4191a880 [ 36.508389] Call trace: [ 36.511582] dev_uevent+0x138/0x1e8 [ 36.515824] uevent_show+0x90/0x118 [ 36.520048] dev_attr_show+0x20/0x58 [ 36.524339] sysfs_kf_seq_show+0xa0/0x110 [ 36.529046] kernfs_seq_show+0x2c/0x9c [ 36.533465] seq_read_iter+0x11c/0x3b4 [ 36.537858] kernfs_fop_read_iter+0x68/0x188 [ 36.542758] vfs_read+0x290/0x2bc [ 36.546666] ksys_read+0x74/0xe0 [ 36.550464] __arm64_sys_read+0x1c/0x28 [ 36.554867] el0_svc_common+0x90/0x110 [ 36.559170] do_el0_svc+0x24/0x80 [ 36.563029] el0_svc+0x28/0x88 [ 36.566622] el0_sync_handler+0x84/0xe4 [ 36.571006] el0_sync+0x154/0x180 [ 36.574856] Code: aa1403e0 97f8c37e f9403668 b40000c8 (f9400102) [ 36.581549] ---[ end trace 556d98a75f645ca9 ]--- [ 36.588678] Unable to handle kernel paging request at virtual address ffff80001087b7d0 [ 36.597285] Mem abort info: [ 36.600580] ESR = 0x96000007 [ 36.604092] EC = 0x25: DABT (current EL), IL = 32 bits [ 36.609969] SET = 0, FnV = 0 [ 36.613503] EA = 0, S1PTW = 0 [ 36.617115] Data abort info: [ 36.620451] ISV = 0, ISS = 0x00000007 [ 36.624759] CM = 0, WnR = 0 [ 36.628167] swapper pgtable: 4k pages, 48-bit VAs, pgdp=00000000013f1000 [ 36.635407] [ffff80001087b7d0] pgd=0000000001810003, p4d=0000000001810003, pud=0000000001811003, pmd=00000000039b9003, pte=0000000000000000 [ 36.649039] Internal error: Oops: 96000007 [#2] PREEMPT SMP [ 36.655146] Modules linked in: mfrc522 hci_uart btqca btbcm bluetooth ecdh_generic ecc 8021q mrp garp stp llc spidev brcmfmac vc4 cec brcmutil drm_kms_helper cfg80211 drm raspberrypi_cpufreq rfkill clk_raspberrypi raspberrypi_hwmon crct10dif_ce i2c_bcm2835 bcm2835_thermal bcm2835_rng rng_core spi_bcm2835 aes_neon_bs aes_neon_blk ip_tables x_tables ipv6 [ 36.688923] CPU: 0 PID: 181 Comm: systemd-udevd Tainted: G D W 5.12.0-rc4+ #25 [ 36.698488] Hardware name: Raspberry Pi 3 Model B Plus Rev 1.3 (DT) [ 36.705416] pstate: 80000005 (Nzcv daif -PAN -UAO -TCO BTYPE=--) [ 36.712093] pc : dev_uevent+0x138/0x1e8 [ 36.716577] lr : uevent_show+0x90/0x118 [ 36.721052] sp : ffff8000104e3ba0 [ 36.724989] x29: ffff8000104e3bc0 x28: ffff670b43875400 [ 36.730970] x27: 0000000000000000 x26: 0000000000000001 [ 36.736956] x25: 0000000000000000 x24: ffff670b47de2e00 [ 36.742944] x23: ffffdc99f37a4168 x22: ffff670b4191a880 [ 36.748923] x21: ffff670b42475800 x20: ffff670b434c0000 [ 36.754895] x19: ffff670b42475800 x18: 0000000000000000 [ 36.760851] x17: 0000000000000000 x16: 0000000000000000 [ 36.766787] x15: ffff670b4342f0d0 x14: 0000000000000000 [ 36.772723] x13: 0000000000000000 x12: 0000000000000000 [ 36.778653] x11: 0000000000000002 x10: 0000000080080000 [ 36.784572] x9 : 0de4e88efe7dd600 x8 : ffff80001087b7d0 [ 36.790498] x7 : 0000000000000000 x6 : 000000000000003f [ 36.796428] x5 : 0000000000000040 x4 : ffff8000104e3b80 [ 36.802356] x3 : 0000000000000001 x2 : ffff670b434c0000 [ 36.808284] x1 : ffff670b42475800 x0 : ffff670b4191a880 [ 36.814202] Call trace: [ 36.817189] dev_uevent+0x138/0x1e8 [ 36.821226] uevent_show+0x90/0x118 [ 36.825254] dev_attr_show+0x20/0x58 [ 36.829352] sysfs_kf_seq_show+0xa0/0x110 [ 36.833874] kernfs_seq_show+0x2c/0x9c [ 36.838115] seq_read_iter+0x11c/0x3b4 [ 36.842337] kernfs_fop_read_iter+0x68/0x188 [ 36.847072] vfs_read+0x290/0x2bc [ 36.850820] ksys_read+0x74/0xe0 [ 36.854464] __arm64_sys_read+0x1c/0x28 [ 36.858717] el0_svc_common+0x90/0x110 [ 36.862874] do_el0_svc+0x24/0x80 [ 36.866589] el0_svc+0x28/0x88 [ 36.870038] el0_sync_handler+0x84/0xe4 [ 36.874278] el0_sync+0x154/0x180 [ 36.877984] Code: aa1403e0 97f8c37e f9403668 b40000c8 (f9400102) [ 36.884527] ---[ end trace 556d98a75f645caa ]--- ``` This crash happens right after returning from our `mfrc522_probe` function. This is quite hard to debug, and is still puzzling us. We had a lot of hope when realizing that we were 30 commits behind on our Rust-for-linux/linux fork, and quite a few of them were referencing PRs regarding `read_iter` and `write_iter` (the main offender in the calltrace). Sadly, upon rebasing, we still had the issue. ### Lack of support for basic numeric enums such as C's enums Rust's enums can be represented as numbers using the `#[repr(u8)]` procedural macro. This allows for some nice abstraction, along the lines of ```rust let enum_value = Mfrc522Register::FifoLevel; let byte_value = enum_value as u8; let final_byte = byte_value << MAGIC_NUMBER & MAGIC_MASK; ``` However, the reverse operation (converting a byte to its enum representation) is not implemented by default. This makes sense, as for example the following code would produce undefined behavior: ```rust= #[repr(u8)] enum Mfrc522Version { Version1 = 0x91, Version2 = 0x92, NotMfrc522 = 0x00, } let byte_value = 0xFFu8; let enum_value = byte_value as Mfrc522Version; // What value of the enum is `enum_value`? ``` However, this makes it really unpractical for embedded developpers, as abstracting the return value of a byte operation is something that happens often, for example when destructuring the byte returned by `Mfrc522Spi::read_command()`, which contains the Power Down mode, the Receiver Mode, and 4 bits referencing the current command. You can implement this yourself, for example using the `From` and `TryFrom` traits. However, this is boilerplate, repetitive and annoying code, especially when the compiler could create the boilerplate for `TryFrom` for you. This way, converting a byte to an enum would return a `Result<EnumType>`, which would need to be handled in order to get the converted enum. To avoid boilerplate code and due to a lack of time, we chose to implement this feature using the dreaded `core::mem::transmute` function, which effectively boils down to a `memcpy` or an abstract C cast. ```c= // Convert a byte to an enum u8 byte_value = 0x91; enum mfrc522_version enum_value; memcpy(&enum_value, &byte_value, 1); // Equivalent to... enum_value = (enum mfrc522_version) byte_value; ``` This is equivalent to the following rust code: ```rust= let byte_value = 0x91u8; let enum_value = unsafe { core::mem::transmute::<u8, Mfrc522Version>(byte_value) }; ``` This is wildly unsafe, and will induce undefined behavior if done improperly. We kept the usage of this to the inside of our driver, and plan to rewrite it soon. You can find the macro in `mfrc522_inner/command.rs`, with the macro `from_byte!()`. ## Advantages ### Integrated Unit testing While we haven't found a way to run the unit tests from the kernel's rust side of things, it was very useful to be able to get some instant feedback while developing the non-alloc parser in Rust. This way, all buffer operations could be tested and verified easily, even on x86 without any knowledge of the MFRC522. We could have enabled the KUnit testing framework, but lacked the time to do so. ### Increased Memory safety Since rust slices have a length, there is no need to pass an extra parameter like in C. This makes for cleaner functions, as well as safer code. For example, the following C function: ```c= int mfrc522_fifo_write(struct spi_device *device, const u8 *buf, u8 len); ``` Became ```rust= pub fn fifo_write(dev: &mut SpiDevice, buf: &[u8]) -> KernelResult; ``` Since the kernel relies heavily on threads and context switching, Rust makes a lot of sense since thread-safety is enforced at compile-time. We encountered issues with that concept multiple times during development, which made us rethink our design and adopt better programming dynamics. For example, the `KernelModule` trait is bounded on the `Sync` and `Send` traits, indicating that any type implementing it should be freely passable between threads, safely. Memory management is also automatic thanks to lifetimes and the borrow checker. This allows our abstractions to constistently be properly unregistered, by implementing the `Drop` trait. This is done by all kernel abstractions, and allow for automatic unregistration (in the case of an SPI driver, or a misc device) and nice constructs tied with ownership. ### Higher level constructs and abstraction than C Since Rust is a higher level programming language than C, we could use all of the abstractions available to us, without falling in a multi-paradigm-hell *C++ style*. Rust allows you to use object oriented programming through traits, as well as functional programming. While functional programming was not really used here apart from __pattern matching__, the traits are really practical and allow the abstractions to produce generic, strongly typed and easy to use code. A good example of this is available in the [`FileOperations` abstraction](https://github.com/Rust-for-Linux/linux/blob/rust/rust/kernel/file_operations.rs), which exposes high level traits to represent common file operations. On top of that, we beneficiate from higher order types, such as `Results` or `Options`, which avoid the use of NULL pointers and allow for clean error handling, with propagation, without relying on C++'s heavy exceptions. As an example, here is a comparison between simplified C parsing and Rust parsing, where pattern matching really shines ```c= if (strcmp(cmd, "mem_read") == 0) return command_new_simple(MEM_READ); else if (strcmp(cmd, "mem_write") == 0) return command_new(MEM_WRITE, arg, arg_len); else if (strcmp(cmd, "get_version") == 0) return command_new_simple(GET_VERSION); else return NULL; ... ``` ```rust= match cmd { "mem_read" => Ok(Command::new_simple(Mfrc522Command::MemRead)), "mem_write" => Ok(Command::new(Mfrc522Command::MemWrite, arg, arg_len)), "get_version" => Ok(Command::new_simple(Mfrc522Command::GetVersion)), _ => Err(...); } ``` ### Good integration with the Kernel When the kernel rust abstractions were written by developers smarter than us, they were quite easy to use and made sense from a Kernel point of view. The integration was simple, intuitive (most of the time) and simply worked. However, developing one of those abstractions safely is an absolute pain, mostly due to the black magic that the kernel produces regularly. ### Smaller codebase As a final point, we also noticed that our Rust codebase was smaller and more concise than our C codebase, for roughly the same amount of features. The github metrics currently describe our repository as 53.6% of C and 42.4% of Rust, with around 100 lines of unit testing not present on the C side.

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.