---
title: Xilinx-XRT-Driver-Fuzzing
author: bynx
tags: fuzzing
---
# Xilinx XRT Linux Interfaces
The XRT platform utilizes two (2) drivers, seperated by privilege:
1. `xocl`: _binds to **user-space** PF1_
1. `xclmgmt`: _binds to **kernel-space** PF0_
<figure>
<center>
<img class=center-cropped />
<figcaption>
<i>Figure 1: Download flow for of an <code>xclbin</code> via Mailbox for `PF0` ➜ `PF1`</i>
</figcaption>
</center>
<style>
.center-cropped {
scale: 0.95;
width: 1000px;
height: 600px;
filter: invert(80%);
background-position: center bottom;
background-repeat: no-repeat;
background-image: url('https://xilinx.github.io/XRT/2020.2/html/_images/sw-mailbox-msd-mpd-download.svg');
}
</style>
</figure>
<!--<img src="linux-usb-subsystem-diagram.svg" />-->
From our perspective, we are specifically interested in _fuzzing `xocl` driver functions from
SW user-space_ (`PF0` ➜ `PF1`).
More explicity, this flow of data only occurs [over software] when the user intends to program
the FPGA compute fabric; XRT uses Dynamic Function Exchange (DFX) to load user-compiled binaries to the **User** partition
The _xclbin_ (image) '(down)load' itself is from the `xclmgmt` driver, which binds to `PF0`:
- `xclmgmt` downloads the bitstream packaged in the bitstream section of _xclbin_ by
programming the ICAP peripheral
- `xclmgmt` then discovers the target frequency of the User partition by reading the _xclbin_
clock section and then programs the clocks which are controlled from Shell
From the user-perspective, the equivalent is:
```bash
/opt/xilinx/xrt/bin/xbutil program --device <bsd> --user /path/to/fpga-img.xclbin
```
## Fuzzer Design Assumptions
Our assumptions:
- [ ] Our input source correctly targets our intended system-under-test (SUT)
(i.e., `xclbin` file ➜ `ioctl` syscall ➜ `icap` peripheral ➜ `...` ➜ `PF0` )
- [ ] No additional security restrictions have been placed beyond default XRT configurations
- _xclbin_ signing, IP rules, other stuff...
- [ ] _xclbin_ files can be constructed using specified syscalls, as defined by XRT
- [ ] `xocl` operates as an standard PCI/FPGA driver w.r.t. kernel-space
- [ ] Fuzzing the `xocl` driver's `ioctl` interface can be performed through an `mpd` plugin
These assumptions then allows us to treat our design ala (linux) kernel fuzzing, giving us our
first design component -- our fuzzer [syzkaller][syzk]
➜ The plan is to extend the current `sys-executer` to operate over Xilinx syscalls
## Syzkaller kernel fuzzer
### Plan
1. Extend the syscall input space to allow fuzzing of _xclbin_ files from `PF1` [<sup>$\dagger$</sup>][syz-newsyscall] [<sup>$\ddagger$</sup>][syz-syscalldef]
3. Launch syzkaller to fuzz the linux kernel, but exlcude irrellevant syscalls
(i.e., focus in on XRT/`xocl` PCIe driver)
1. Backup: Use `xoxl` struct and function signatures to hook in a dumb fuzzer for `ioctl`
<sup>
<br>
[_<sup>$\dagger$</sup>_ _Describing new system calls_][syz-newsyscall]
[_<sup>$\ddagger$</sup>_ _Syscall description language_][syz-syscalldef]
</sup>
#### Example Extension Syntax
```c
/* Example of sys-execute syscall syntax */
/*
* https://github.com/google/syzkaller/blob/master/sys/openbsd/dev_pci.txt
*/
include <sys/types.h>
include <sys/pciio.h>
include <fcntl.h>
resource fd_pci[fd]
openat$pci(fd const[AT_FDCWD], file ptr[in, string["/dev/pci"]], flags flags[open_flags], mode const[0]) fd_pci
ioctl$PCIOCREAD(fd fd_pci, cmd const[PCIOCREAD], arg ptr[out, pci_io])
ioctl$PCIOCWRITE(fd fd_pci, cmd const[PCIOCWRITE], arg ptr[in, pci_io])
ioctl$PCIOCGETROM(fd fd_pci, cmd const[PCIOCGETROM], arg ptr[out, pci_rom])
pci_io {
pi_sel pcisel
pi_reg int32
pi_width int32
pi_data int32
}
pci_rom {
pr_sel pcisel
pr_romlen len[pr_rom, int32]
pr_rom ptr[out, array[int32]]
}
pcisel {
pc_bus int8
pc_dev int8
pc_func int8
}
/*
* https://github.com/google/syzkaller/blob/master/sys/openbsd/dev_pci.txt.const
*/
# Code generated by syz-sysgen. DO NOT EDIT.
arches = amd64
AT_FDCWD = amd64:18446744073709551516
PCIOCGETROM = amd64:3222302725
PCIOCREAD = amd64:3222302722
PCIOCWRITE = amd64:3222302723
SYS_ioctl = amd64:54
SYS_openat = amd64:321
```
## Harness Construction: Execution Flow Tracing
1. `xbutil program --device <bsd> --user /path/to/hello-world.xclbin`
- [src fn][xbutil-program] corresponding to: `xbutil program <...>`
```c++
/*
* xbutil program --device <bsd> --user /path/to/hello-world.xclbin
*/
int program(const std::string& xclbin, unsigned region) {
std::ifstream stream(xclbin.c_str());
if(!stream.is_open()) {
std::cout
<< "ERROR: Cannot open "
<< xclbin
<< ". Check that it exists and is readable."
<< std::endl;
return -ENOENT;
}
if(region) {
std::cout << "ERROR: Not support other than -r 0 " << std::endl;
return -EINVAL;
}
char temp_8_;
stream.read(temp, 8);
if (std::strncmp(temp, "xclbin0", 8)) {
if (std::strncmp(temp, "xclbin2", 8))
return -EINVAL;
}
stream.seekg(0, stream.end);
int length = stream.tellg();
stream.seekg(0, stream.beg);
std::vector<char> buffer(length);
stream.read(buffer.data(), length);
auto header = reinterpret_cast<const xclBin *>(buffer.data());
int result = xclLoadXclBin(m_handle, header); // !!!!!!!!!
return result;
}
```
2. `xclLoadXclBin(_,_)` [source][xclload]
```c++
/*
* Parent: int result = xclLoadXclBin(m_handle, header);
*/
int shim::xclLoadXclBin(const xclBin *buffer)
{
auto top = reinterpret_cast<const axlf*>(buffer);
auto ret = xclLoadAxlf(top); // !!!!!!!!!
if (ret != 0) {
if (ret == -EOPNOTSUPP) {
xrt_logmsg(XRT_ERROR, "Xclbin does not match shell on card.");
auto xclbin_vbnv = xrt_core::xclbin::get_vbnv(top);
auto shell_vbnv = xrt_core::device_query<xrt_core::query::rom_vbnv>(mCoreDevice);
if (xclbin_vbnv != shell_vbnv) {
xrt_logmsg(XRT_ERROR, "Shell VBNV is '%s'", shell_vbnv.c_str());
xrt_logmsg(XRT_ERROR, "Xclbin VBNV is '%s'", xclbin_vbnv.c_str());
}
xrt_logmsg(XRT_ERROR, "Use 'xbmgmt flash' to update shell.");
}
else if (ret == -EBUSY) {
xrt_logmsg(XRT_ERROR, "Xclbin on card is in use, can't change.");
}
else if (ret == -EKEYREJECTED) {
xrt_logmsg(XRT_ERROR, "Xclbin isn't signed properly");
}
else if (ret == -E2BIG) {
xrt_logmsg(XRT_ERROR, "Not enough host_mem for xclbin");
}
else if (ret == -ETIMEDOUT) {
xrt_logmsg(XRT_ERROR,
"Can't reach out to mgmt for xclbin downloading");
xrt_logmsg(XRT_ERROR,
"Is xclmgmt driver loaded? Or is MSD/MPD running?");
}
else if (ret == -EDEADLK) {
xrt_logmsg(XRT_ERROR, "CU was deadlocked? Hardware is not stable");
xrt_logmsg(XRT_ERROR, "Please reset device with 'xbutil reset'");
}
xrt_logmsg(XRT_ERROR, "See dmesg log for details. err=%d", ret);
}
return ret;
}
```
3. `xclLoadAxlf(_,_)` [source][xclload2]
```c++
/*
* Parent: auto ret = xclLoadAxlf(top);
*/
int shim::xclLoadAxlf(const axlf *buffer)
{
xrt_logmsg(XRT_INFO, "%s, buffer: %s", __func__, buffer);
drm_xocl_axlf axlf_obj = {const_cast<axlf *>(buffer), 0};
int off = 0;
auto kernels = xrt_core::xclbin::get_kernels(buffer);
/* Calculate size of kernels */
for (auto& kernel : kernels) {
axlf_obj.ksize += sizeof(kernel_info) + sizeof(argument_info) * kernel.args.size();
}
/* To enhance CU subdevice and KDS/ERT, driver needs all details about kernels
* while load xclbin.
*
* Why we extract data from XML metadata?
* 1. Kernel is NOT a good place to parse xml. It prefers binary.
* 2. All kernel details are in the xml today.
*
* What would happen in the future?
* XCLBIN would contain fdt as metadata. At that time, this
* could be removed.
*
* Binary format:
* +-----------------------+
* | Kernel[0] |
* | name[64] |
* | anums |
* | argument[0] |
* | argument[1] |
* | argument[...] |
* |-----------------------|
* | Kernel[1] |
* | name[64] |
* | anums |
* | argument[0] |
* | argument[1] |
* | argument[...] |
* |-----------------------|
* | Kernel[...] |
* | ... |
* +-----------------------+
*/
std::vector<char> krnl_binary(axlf_obj.ksize);
axlf_obj.kernels = krnl_binary.data();
for (auto& kernel : kernels) {
auto krnl = reinterpret_cast<kernel_info *>(axlf_obj.kernels + off);
if (kernel.name.size() > sizeof(krnl->name))
return -EINVAL;
std::strncpy(krnl->name, kernel.name.c_str(), sizeof(krnl->name)-1);
krnl->name[sizeof(krnl->name)-1] = '\0';
krnl->anums = kernel.args.size();
krnl->range = kernel.range;
int ai = 0;
for (auto& arg : kernel.args) {
if (arg.name.size() > sizeof(krnl->args[ai].name)) {
xrt_logmsg(XRT_ERROR, "%s: Argument name length %d>%d", __func__, arg.name.size(), sizeof(krnl->args[ai].name));
return -EINVAL;
}
std::strncpy(krnl->args[ai].name, arg.name.c_str(), sizeof(krnl->args[ai].name)-1);
krnl->args[ai].name[sizeof(krnl->args[ai].name)-1] = '\0';
krnl->args[ai].offset = arg.offset;
krnl->args[ai].size = arg.size;
// XCLBIN doesn't define argument direction yet and it only support
// input arguments.
// Driver use 1 for input argument and 2 for output.
// Let's refine this line later.
krnl->args[ai].dir = 1;
ai++;
}
off += sizeof(kernel_info) + sizeof(argument_info) * kernel.args.size();
}
/* To make download xclbin and configure KDS/ERT as an atomic operation. */
axlf_obj.kds_cfg.ert = xrt_core::config::get_ert();
axlf_obj.kds_cfg.polling = xrt_core::config::get_ert_polling();
axlf_obj.kds_cfg.cu_dma = xrt_core::config::get_ert_cudma();
axlf_obj.kds_cfg.cu_isr = xrt_core::config::get_ert_cuisr() && xrt_core::xclbin::get_cuisr(buffer);
axlf_obj.kds_cfg.cq_int = xrt_core::config::get_ert_cqint();
axlf_obj.kds_cfg.dataflow = xrt_core::config::get_feature_toggle("Runtime.dataflow") || xrt_core::xclbin::get_dataflow(buffer);
axlf_obj.kds_cfg.rw_shared = xrt_core::config::get_rw_shared();
/* TODO: In scheduler.cpp init() function, it use get_ert_slots(void) to get slot size.
* But we cannot do this here, since the xclbin is not registered.
* Currently, emulation flow use get_ert_slots() as well.
* We will consider how to better determine slot size in new kds.
*/
//axlf_obj.kds_cfg.slot_size = mCoreDevice->get_ert_slots().second;
auto xml_hdr = xrt_core::xclbin::get_axlf_section(buffer, EMBEDDED_METADATA);
if (!xml_hdr)
throw std::runtime_error("No xml metadata in xclbin");
auto xml_size = xml_hdr->m_sectionSize;
auto xml_data = reinterpret_cast<const char*>(reinterpret_cast<const char*>(buffer) + xml_hdr->m_sectionOffset);
axlf_obj.kds_cfg.slot_size = mCoreDevice->get_ert_slots(xml_data, xml_size).second;
int ret = mDev->ioctl(mUserHandle, DRM_IOCTL_XOCL_READ_AXLF, &axlf_obj); // !!!!!!!!!
if(ret)
return -errno;
// If it is an XPR DSA, zero out the DDR again as downloading the XCLBIN
// reinitializes the DDR and results in ECC error.
if(isXPR())
{
xrt_logmsg(XRT_INFO, "%s, XPR Device found, zeroing out DDR again..", __func__);
if (zeroOutDDR() == false)
{
xrt_logmsg(XRT_ERROR, "%s, zeroing out DDR again..", __func__);
return -EIO;
}
}
return ret;
}
```
4. This leaves us at the raw `ioctl` [syscall][ioctl-shim]:
```c++
/*
* FnSig: int ioctl(int fd, cmd const[DRM_IOCTL_XOCL], const axlf* buf)
*/
int ret = mDev->ioctl(mUserHandle, DRM_IOCTL_XOCL_READ_AXLF, &axlf_obj);
```
## Appendix A: Quick Definitions
### **_xclbin_**
- **_What is it?_**
➜ A contained which packs FPGA bitstreams and collections of metadata (i.e., memory
topology, IP instantiations, etc).
- **_What does it look like?_**
➜ [_xclbin_ format definition][xclbin-fmt]
#### **_Mailbox Subdevice Drive (`msd`)_**
- **_What is it?_**
➜ A link between two (2), otherwise separate, processor systems
- **_What does it do?_**
➜ Bi-directional IPC; Generate interrupts between the processors
- **_What does it look like?_**
➜ [mailbox_msg][mailbox-msg] and [mailbox_protocol][mailbox-proto]
(`src/runtime_src/core/pcie/driver/linux/xocl/subdev/mailbox.c`)
### **_xoxl_**
> Xilinx® PCIe platforms like Alveo PCIe cards support various memory topologies which
> can be dynamically loaded as part of FPGA image loading step. This means from one
> FPGA image to another the device may expose one or more memory controllers where
> each memory controller has its own memory address range. We use Linux `_drm_mm_` for
> allocation of memory and Linux _drm_gem_ framework for mmap handling. Since ordinarily
> our device memory is not exposed to host CPU (except when we enable PCIe peer-to-peer
> feature) we use host memory pages to back device memory for mmap support. For
> syncing between device memory and host memory pages XDMA/QDMA PCIe memory
> mapped DMA engine is used. Users call sync ioctl to effect DMA in requested direction.
[xocl execution model][xocl-execmodel]
### **_xclmgmt_**
> XRT Linux kernel driver _xclmgmt_ binds to management physical function. Management
> physical function provides access to Shell components responsible for **privileged**
> operations. xclmgmt driver is organized into subdevices and handles the following
> functionality:
### **_data flow breakdown_**
A typical **user** execution flow (via **PF1**):
> 1. **Load _xclbin_** using `DOWNLOAD` `ioctl`
> 1. Discover compute unit register map from xclbin
> 1. **Allocate data buffers** to feed compute units using `CREATE_BO/MAP_BO` `ioctl` calls
> 1. Migrate input data buffers from host to device using `SYNC_BO` `ioctl`
> 1. **Allocate an execution command buffer** using `CREATE_BO/MAP_BO` `ioctl` call and fill
> command buffer using data in 2 above and following the format defined in `ert.h`
> 1. **Submit the execution command buffer** using `EXECBUF` `ioctl`
> 1. Wait for completion using POSIX `poll`
> 1. Migrate output data buffers from device to host using `SYNC_BO` `ioctl`
> 1. **Release data buffers and command buffer**
### misc but helpful probs
- `xclbin_parser` [source][xclbin-parser]
- software channel of mailbox [explained][mailbox-src-sw]
- `msd` plugin [example src][msd-plug]
- [linux-xlnx][nix-xlnx] PCIe controller functions for Xilinix Linux Kernel
- [custom xclbin protections][xrt-container]
[xbutil-program]: https://github.com/Xilinx/XRT/blob/master/src/runtime_src/core/pcie/tools/xbutil/xbutil.h#L1768-L1803
[xclload]: https://github.com/Xilinx/XRT/blob/master/src/runtime_src/core/pcie/linux/shim.cpp#L1308-L1349
[xclload2]: https://github.com/Xilinx/XRT/blob/master/src/runtime_src/core/pcie/linux/shim.cpp#L1351-L1467
[ioctl-shim]: https://github.com/Xilinx/XRT/blob/master/src/runtime_src/core/edge/user/shim.cpp#L578
[icap-user]:https://github.com/Xilinx/XRT/blob/master/src/runtime_src/core/pcie/driver/linux/xocl/subdev/icap.c#L2388-L2453
[icap-kernel]: https://github.com/Xilinx/XRT/blob/master/src/runtime_src/core/pcie/driver/linux/xocl/subdev/icap.c#L2455-L2530
[icap-shared]: https://github.com/Xilinx/XRT/blob/master/src/runtime_src/core/pcie/driver/linux/xocl/subdev/icap.c#L2532-L2629
[syzk]: https://github.com/google/syzkaller/
[syz-newsyscall]: https://github.com/google/syzkaller/blob/master/docs/syscall_descriptions.md#describing-new-system-calls
[syz-syscalldef]: https://github.com/google/syzkaller/blob/master/docs/syscall_descriptions_syntax.md
[xclbin-fmt]: https://xilinx.github.io/XRT/2020.2/html/formats.html#formats-rst
[mailbox-msg]: https://github.com/Xilinx/XRT/blob/master/src/runtime_src/core/pcie/driver/linux/xocl/subdev/mailbox.c#L281-L299
[mailbox-proto]: https://github.com/Xilinx/XRT/blob/master/src/runtime_src/core/pcie/driver/linux/include/mailbox_proto.h
[xocl-execmodel]: https://xilinx.github.io/XRT/2020.2/html/execution-model.html#xocl
[xclbin-parser]: https://github.com/Xilinx/XRT/blob/master/src/runtime_src/core/common/xclbin_parser.cpp
[mailbox-src-sw]: https://github.com/Xilinx/XRT/blob/master/src/runtime_src/core/pcie/driver/linux/xocl/subdev/mailbox.c#L142
[msd-plug]: https://github.com/Xilinx/XRT/blob/master/src/runtime_src/core/pcie/tools/cloud-daemon/msd_plugin_example.c
[nix-xlnx]: https://github.com/Xilinx/linux-xlnx/blob/master/drivers/pci/controller/pcie-xilinx.c#L96
[xrt-container]: https://github.com/Xilinx/XRT/blob/master/src/runtime_src/core/pcie/tools/cloud-daemon/container/container.cpp#L167
Sign in with Wallet
Connect another wallet