# `struct dma_fence` notes # tl;dr `struct dma_fence` is refcounted. It must be initialized after allocation, which will leave the object with a refcount of 1. When the refcount becomes zero the object is freed. The default implementation will free lazily with `kfree_rcu()`. The structure can be embedded in other structures. If embedded as the first field, the refcounting of the embedded object can be used to manage the lifetime of the super object. This is the intended use. The structure is intended to be dynamically allocated, although it might work stack allocated if the original owner never decrements the refcount. The fence can be customized by populating a vtable. Most notable is the option to change the cleanup function that is called when the refcount becomes zero, and the default wait function. The primary function provided by `struct dma_fence` is that of a semaphore. Multiple waiters can wait on the fence, to be released when the fence is signaled. As an additional feature, users of the fence can enqueue callbacks to be called when the fence is signaled. Waiters of the fence must obtain a reference on the fence before waiting on it. Methods for obtaining refcounts in normal and RCU context are provided. All operations on the fence is protected by a spinlock. This lock is provided by the user at initialization time as a pointer that is installed in the object. # Details ## Files * `include/linux/dma-fence.h` * `drivers/dma-buf/dma-fence.c` * `drivers/gpu/drm/virtio/virtgpu_drv.h` * `drivers/gpu/drm/virtio/virtgpu_fence.c` * `include/drm/drm_syncobj.h` * `drivers/gpu/drm/drm_syncobj.c` ## Description ### Lifted from `dma-fence.c` DOC: DMA fences overview DMA fences, represented by &struct dma_fence, are the kernel internal synchronization primitive for DMA operations like GPU rendering, video encoding/decoding, or displaying buffers on a screen. A fence is initialized using dma_fence_init() and completed using dma_fence_signal(). Fences are associated with a context, allocated through dma_fence_context_alloc(), and all fences on the same context are fully ordered. Since the purposes of fences is to facilitate cross-device and cross-application synchronization, there's multiple ways to use one: - Individual fences can be exposed as a &sync_file, accessed as a file descriptor from userspace, created by calling sync_file_create(). This is called explicit fencing, since userspace passes around explicit synchronization points. - Some subsystems also have their own explicit fencing primitives, like &drm_syncobj. Compared to &sync_file, a &drm_syncobj allows the underlying fence to be updated. - Then there's also implicit fencing, where the synchronization points are implicitly passed around as part of shared &dma_buf instances. Such implicit fences are stored in &struct dma_resv through the &dma_buf.resv pointer. DOC: fence cross-driver contract Since &dma_fence provide a cross driver contract, all drivers must follow the same rules: * Fences must complete in a reasonable time. Fences which represent kernels and shaders submitted by userspace, which could run forever, must be backed up by timeout and gpu hang recovery code. Minimally that code must prevent further command submission and force complete all in-flight fences, e.g. when the driver or hardware do not support gpu reset, or if the gpu reset failed for some reason. Ideally the driver supports gpu recovery which only affects the offending userspace context, and no other userspace submissions. * Drivers may have different ideas of what completion within a reasonable time means. Some hang recovery code uses a fixed timeout, others a mix between observing forward progress and increasingly strict timeouts. Drivers should not try to second guess timeout handling of fences from other drivers. * To ensure there's no deadlocks of dma_fence_wait() against other locks drivers should annotate all code required to reach dma_fence_signal(), which completes the fences, with dma_fence_begin_signalling() and dma_fence_end_signalling(). * Drivers are allowed to call dma_fence_wait() while holding dma_resv_lock(). This means any code required for fence completion cannot acquire a &dma_resv lock. Note that this also pulls in the entire established locking hierarchy around dma_resv_lock() and dma_resv_unlock(). * Drivers are allowed to call dma_fence_wait() from their &shrinker callbacks. This means any code required for fence completion cannot allocate memory with GFP_KERNEL. * Drivers are allowed to call dma_fence_wait() from their &mmu_notifier respectively &mmu_interval_notifier callbacks. This means any code required for fence completeion cannot allocate memory with GFP_NOFS or GFP_NOIO. Only GFP_ATOMIC is permissible, which might fail. Note that only GPU drivers have a reasonable excuse for both requiring &mmu_interval_notifier and &shrinker callbacks at the same time as having to track asynchronous compute work using &dma_fence. No driver outside of drivers/gpu should ever call dma_fence_wait() in such contexts. ### Structure Layout ```c /** * struct dma_fence - software synchronization primitive * @refcount: refcount for this fence * @ops: dma_fence_ops associated with this fence * @rcu: used for releasing fence with kfree_rcu * @cb_list: list of all callbacks to call * @lock: spin_lock_irqsave used for locking * @context: execution context this fence belongs to, returned by * dma_fence_context_alloc() * @seqno: the sequence number of this fence inside the execution context, * can be compared to decide which fence would be signaled later. * @flags: A mask of DMA_FENCE_FLAG_* defined below * @timestamp: Timestamp when the fence was signaled. * @error: Optional, only valid if < 0, must be set before calling * dma_fence_signal, indicates that the fence has completed with an error. * * the flags member must be manipulated and read using the appropriate * atomic ops (bit_*), so taking the spinlock will not be needed most * of the time. * * DMA_FENCE_FLAG_SIGNALED_BIT - fence is already signaled * DMA_FENCE_FLAG_TIMESTAMP_BIT - timestamp recorded for fence signaling * DMA_FENCE_FLAG_ENABLE_SIGNAL_BIT - enable_signaling might have been called * DMA_FENCE_FLAG_USER_BITS - start of the unused bits, can be used by the * implementer of the fence for its own purposes. Can be used in different * ways by different fence implementers, so do not rely on this. * * Since atomic bitops are used, this is not guaranteed to be the case. * Particularly, if the bit was set, but dma_fence_signal was called right * before this bit was set, it would have been able to set the * DMA_FENCE_FLAG_SIGNALED_BIT, before enable_signaling was called. * Adding a check for DMA_FENCE_FLAG_SIGNALED_BIT after setting * DMA_FENCE_FLAG_ENABLE_SIGNAL_BIT closes this race, and makes sure that * after dma_fence_signal was called, any enable_signaling call will have either * been completed, or never called at all. */ struct dma_fence { spinlock_t *lock; const struct dma_fence_ops *ops; /* * We clear the callback list on kref_put so that by the time we * release the fence it is unused. No one should be adding to the * cb_list that they don't themselves hold a reference for. * * The lifetime of the timestamp is similarly tied to both the * rcu freelist and the cb_list. The timestamp is only set upon * signaling while simultaneously notifying the cb_list. Ergo, we * only use either the cb_list of timestamp. Upon destruction, * neither are accessible, and so we can use the rcu. This means * that the cb_list is *only* valid until the signal bit is set, * and to read either you *must* hold a reference to the fence, * and not just the rcu_read_lock. * * Listed in chronological order. */ union { struct list_head cb_list; /* @cb_list replaced by @timestamp on dma_fence_signal() */ ktime_t timestamp; /* @timestamp replaced by @rcu on dma_fence_release() */ struct rcu_head rcu; }; u64 context; u64 seqno; unsigned long flags; struct kref refcount; int error; }; ``` ## Use `dma_fence` is a synchronization primitive. It's function can be customized by populating a vtable. In the default configuration, after fence allocation and initialization, threads can wait for the fence to be signaled. Once the fence is signaled, all waiters re notified. The fence is one-shot. Before waiting on the fence, callers must obtain a reference. This can be done with `dma_fence_get()` in normal context or `dma_fence_get_rcu()` in rcu read side context. ## Implementation `dma_fence`is implemented with a spinlock pointer (`fence->lock`) and a callback list. The callback list is a linked list of: ```c struct dma_fence_cb { struct list_head node; dma_fence_func_t func; }; ``` Callbacks are added to the list through the default wait function `dma_fence_default_wait()` which will add `dma_fence_default_wait_cb()` to the callback list, or through `dma_fence_add_callback()` which will add an arbitrary callback function to the list. > ⚠️ Callbacks cannot be added after the fence has been signaled. This will > result in `-ENOENT`. Entries are added to the list with `fence->lock` held. The default wait function `dma_fence_default_wait()` (activated by `dma_fence_wait()` through the vtable) will put the function `dma_fence_default_wait_cb` on the callback list. This function wakes up the task that called `dma_fence_default_wait()`. After queuing the callback, `dma_fence_default_wait()` goes to sleep by `schedule_timeout()`. On wake, the task will grab `fence->lock` again and check that the fence was signaled. If not, an error is returned. The caller of `dma_fence_wait()` must hold a reference to the fence before the call. To signal the fence, a user has to call one of the signaling methods, the simplest being `dma_fence_signal()`. This will acquire `fence->lock`, take the callback list, call each callback, and release the lock. The callback list of `struct dma_fence` is aliased with two other fields. After initialization, the location of the callback list is a `struct list_head`. After signaling, it becomes a `ktime_t` that stores the timestamp when the fence was signaled. When `dma_fence_release()` is called, the location becomes a `struct rcu_head` used for cleaning up. ### Initialization and cleanup `struct dma_fence` is reference counted by embedded `struct kref`. `struct dma_fence` has to be initialized by a call to `dma_fence_init()`. This will initialize all fields of the structure, such as lock, refcount, vtable, etc. After `dma_fence_init()`, the fence will have a refcount of 1. The fence is cleaned up when the refcount goes to zero by a call to `dma_fence_release()`. This will in turn call either a custom clean up function via the vtable, or the default `dma_fence_free()`, which calls `kfree_rcu`. This means that if the structure is stack allocated, the refcount must _never_ go to zero and the owner must never call `dma_fence_put()`. The object must have been allocated with `kmalloc()` or `kmem_cache_alloc()`. If the structure is embedded within another structure, `fence->ops->release` _must_ be customized if the `struct dma_fence` is not the first field of the parent structure. ## Embedding `struct virtio_gpu_fence` is an example of embedding `struct dma_fence` for the purpose of customizing behavior. This driver uses batching of fence signaling. To queue an event for emission, the driver will add a `struct virtio_gpu_fence` to a list. Later, it will signal all of the fences on the list that are ready. The driver uses one lock for all its fences. Fences are freed through RCU by the default clean up function once refcount goes to zero.