2020 FreeBSD Foundation Software Development

Wow, what a year 2020's been. Despite the challenges of the global pandemic, the Foundation's software development efforts proceeded apace, and we've made significant contributions across a number of functional areas in FreeBSD.

Since the beginning of 2020 in the FreeBSD src repository we've had 838XXX commits tagged as being sponsored by the FreeBSD Foundation, across 12 different authors. This is a little over 30% of the commits that identify a sponsor. This includes work by project grant recipients, by the Foundation's own staff, as well commits made on behalf of the 6 University of Waterloo co-operative education students who worked for the Foundation over two terms during the year. In addition, and in contrast with previous years many more contributions are now being made to third-party projects under Foundation sponsorship. This includes LLVM's LLDB debugger, OpenBroadcaster-Studio software, Firefox, and others.

Here I'll recap the Foundation-sponsored work started and completed over the past year.

Sponsored projects

OpenZFS Zstd

We sponsored Allan Jude to integrate Zstd compression support into OpenZFS. This is work that Allan started as a prototype years ago, with contributions and updates from others over that time. This project involved completing outstanding tasks to bring the work to a mergeable state, adding tests, and benchmarking. This work initially targeted the OpenZFS repository, and was brought into FreeBSD as part of an OpenZFS update.

Linuxulator application compatibility

Edward Napierała worked on Linux Emulation Layer (Linuxulator) improvements over the years, under Foundation sponsorship. In 2020 we sponsored Edward to continue improving the Linuxulator, but taking a different approach. Earlier grants focused on improving debugging and diagnostic tools, general infrastructure improvements, and similar tasks. In contrast, this time we focused on identifying a number of interesting server and client Linux applications. We identifed any missing required functionality while trying to run them, and fixed many issues found as a result. When we started this work contemporary Linux binaries did not even start correctly, and now it is possible to run a substantial number of them. Others in the FreeBSD community have built upon Edward's work, investigating issues with configuration, packaging, installation and such, and provided guides or sketches for running various programs on FreeBSD.

Contemporary Graphics Driver Support

Emmanuel Vadot worked on improving and updating the graphics (DRM) device drivers, bringing us to parity with device support in Linux kernel version 5.4. The project also made progress on the BSD-licensed implementation of the Linux KPI compatibility layer, with the intent of moving the graphics drivers back into the base system once the work is complete. The work also included a backlight control framework and backlight(8) control program. Some portions of this work were completed in the FreeBSD base system (src) repository, while others were part of the drm-kmod FreeBSD port.

Bridge performance improvements

The Foundation funded Kristof Provost to address bottlenecks in if_bridge(4), the network bridge device. The project was completed early in the year, delivering around a 5X throughput increase. The work is described more fully in XXX.

LLDB

XXX https://github.com/freebsd/freebsd-quarterly/pull/279/commits/951e3f47b83b5fa668725636ca89ba005d1905ff
We funded Kamil Rytarowski and Michał Górny of Moritz Systems to update LLDB's FreeBSD support, triage test suite failures, and fix bugs. Before this work LLDB's FreeBSD support relied on an older, same-process model, and was the only operating system to do so. This project ported the x86 FreeBSD target support to the new debugserver model. This work has been committed to the LLDB repository in advance of LLDB 12.0's release. The work also identified issues in the FreeBSD base system (kernel and runtime components) that have now been fixed.

Test Lab intfrastructure

John-Mark Gurney is developing a management interface for remote access to hardware test labs, to allow reserving a device in the lab, controlling power, providing console access, and managing network-booted root filesystems.

WiFi 802.11ac and driver support

Björn Zeeb continued work on WiFi infrastructure improvements. This work is focused on updating the net80211 stack for 802.11ac support, and allowing existing dual-licensed kernel drivers to be used on FreeBSD. Björn has been collaborating with upstream driver developers with a goal of submitting patches to ease our maintenance burden.
XXX bz will notes from bz in quarterly

RAID-Z expansion

Our long-running RAID-Z expansion project continued this year, including a merge with the upstream OpenZFS with dRAID support (which some work developed as part of this project). Additional contributions came from community members ,including additional tests and optimized math routines.

Co-op

The Foundation hired six co-op students from the University of Waterloo during the 2020 winter and fall terms, as well as one intern XXX. You can read more about Colin, Tiger, and Yang's work (the first three students) in earlier Foundation reports. For the fall term (which just wrapped up) Tiger returned, and new students Yang and Zac joined us for the first time.

Tiger worked on improvements to the code-coverage guided kernel fuzzing tool Syzkaller, adding new system call definitions so that Syzkaller can expand the code it tests. A number of FreeBSD kernel bug fixes have already resulted from this work. Tiger also contributed a number of improvements to the ELF Tool Chain set of binary utilities, and worked on tooling to run tests from other tool suites against ELF Tool Chain.

Zac worked on an improvement to the pkg package management tool, investigating and upstreaming patches for FreeBSD support in FreePBX, and investigating compiler support for addressing the stack clash vulnerability.

Yang investigated and fixed a compilation bug with the kernel's Skein-1024 assembly implementation (used by ZFS), and then a number of projects related to Capsicum: applying Capsicum to sort(1), implementing a Capsicum service to execute utilities, and finally working with developers of the Game of Trees (got) version control system to adapt it for Capsicum support.

XXX: added by lwhsu
Ka Ho focused on improving the desktop experience of the FreeBSD. He fixed and improved many places of OBS (Open Broadcaster Software) on FreeBSD, worked on FreeBSD native audio support on Firefox, adding a facility that user-space audio programs could make use of to enumerate a list of audio devices. He also ported the fcitx5 input method framework.

Staff

The five Foundation staff members continued contributions in 2020 in both ongoing operational tasks and software development for a number of projects.

Staff members responded to reported security vulnerabilities and release errata, prepared patches, and participated in the security advisory process. We also worked on proactive security vulnerability mitigations. Syzkaller also provided many reports of kernel issues that resulted in Foundation-sponsored bug fixes.

We participated in code reviews and supported community members in integrating changes into FreeBSD, and triaged incoming bug reports.

arm64

• linuxulator
• vmware/uhci
• hwcaps
• ifunc, atomics, stack capture, LSE
• crypto

We contributed enhancements to many kernel and userland subsystems, including the x86 pmap layer, ELF run-time linker and kernel loader, the Capsicum sandboxing framework and Casper services, the threading library, some RISC-V changes, the build system, tool chain and freebsd-update, network stack stability improvements, machine-dependent optimizations, new kernel interfaces, DTrace bug fixes, documentation improvements, and others.

XXX CI

• Our own CI
• 3rd party CI integration