[Note] A glimpse of Workqueue in Linux

[Note] A glimpse of Workqueue in Linux === ###### tags: `note` `linux` `workqueue` `CMWQ(Concurrency Managed Workqueue)` `LKM` `timer` `DPC` `bottom half` `events` [TOC] ## Preface Workqueue is generally used in device drivers. In kernel vesion prior to 2.6.36, the implementation of workqueue is simple (about 800 lines). However, in kernel 2.6.36, CMWQ(Concurrency Managed Workqueue) is introduced and becomes more complicated over 5000 lines. Below figure shows structures and relation in CMWQ. ![](https://i.imgur.com/oNyx0Zh.png)[^ref6] ## Introducing work queues[^ref1] > Work queues are a more recent deferral mechanism, added in the 2.5 Linux kernel version. Rather than providing a one-shot deferral scheme as is the case with tasklets, work queues are a generic deferral mechanism in which the handler function for the work queue can sleep (not possible in the tasklet model). Work queues can have higher latency than tasklets but include a richer API for work deferral. Deferral used to be managed by task queues through keventd but is now managed by kernel worker threads named events/X (X is the number of CPU begining from 0). Work queues provide a generic method to defer functionality to bottom halves. At the core is the work queue (struct workqueue_struct), which is the structure onto which work is placed. Work is represented by a work_struct structure, which identifies the work to be deferred and the deferral function to use (see Figure 3). The events/X kernel threads (one per CPU) extract work from the work queue and activates one of the bottom-half handlers (as indicated by the handler function in the struct work_struct).<center> ![](https://i.imgur.com/BZ2HCEd.png) Figure 3. The process behind work queues</center> As the work_struct indicates the handler function to use, you can use the work queue to queue work for a variety of handlers. Now, let's look at the API functions that can be found for work queues. ## Synopsis[^ref3][^ref4][^ref5] | API | description | |:------ |:----------- | |<tr><td colspan="2">Specific workqueues operation</td></tr> |[**create_workqueue**](https://elixir.bootlin.com/linux/latest/source/include/linux/workqueue.h#L448)|allocate a workqueue| |[**INIT_WORK** **INIT_DELAYED_WORK**](https://elixir.bootlin.com/linux/latest/source/include/linux/workqueue.h#L222)|initialize all of a work item in one go |[**queue_work**](https://www.fsl.cs.sunysb.edu/kernel-api/re54.html)|queue work on a workqueue| |[**queue_delayed_work**](https://www.fsl.cs.sunysb.edu/kernel-api/re55.html)|queue work on a workqueue after delay| |[**queue_delayed_work_on**](https://www.fsl.cs.sunysb.edu/kernel-api/re56.html)|queue work on specific CPU after delay| |[**flush_workqueue**](https://www.fsl.cs.sunysb.edu/kernel-api/re57.html)|ensure that any scheduled work has run to completion.| |[**destroy_workqueue**](https://www.fsl.cs.sunysb.edu/kernel-api/re58.html)|safely terminate a workqueue| |<tr><td colspan="2">Kernel-global workqueues operation</td></tr> |[**schedule_work**](https://www.fsl.cs.sunysb.edu/kernel-api/re59.html)|put work task in global workqueue| |[**schedule_delayed_work**](https://www.fsl.cs.sunysb.edu/kernel-api/re60.html)|put work task in global workqueue after delay| |[**schedule_delayed_work_on**](https://www.fsl.cs.sunysb.edu/kernel-api/re61.html)|queue work in global workqueue on CPU after delay| |<tr><td colspan="2">Work operation</td></tr> |[**flush_work**](http://www.hep.by/gnu/kernel/device-drivers/API-flush-work.html)|wait for a work to finish executing the last queueing instance| |[**cancel_work_sync**](http://www.hep.by/gnu/kernel/device-drivers/API-cancel-work-sync.html)|cancel a work and wait for it to finish| |[**flush_delayed_work**](http://www.hep.by/gnu/kernel/device-drivers/API-flush-delayed-work.html)|wait for a dwork to finish executing the last queueing| |[**cancel_delayed_work**](https://linuxtv.org/downloads/v4l-dvb-internals/device-drivers/API-cancel-delayed-work.html)|cancel a delayed work| |[**cancel_delayed_work_sync**](http://www.hep.by/gnu/kernel/device-drivers/API-cancel-delayed-work-sync.html)|cancel a delayed work and wait for it to finish| |[**work_pending**](https://elixir.bootlin.com/linux/latest/source/include/linux/workqueue.h#L290)|find out whether a work item is currently pending |[**delayed_work_pending**](https://elixir.bootlin.com/linux/latest/source/include/linux/workqueue.h#L297)| find out whether a delayable work item is currently ## Workqueue API[^ref2] ```C= //v4.18.14 - include/linux/workqueue.h // //Work initialization macros to bind work with func // #define INIT_DELAYED_WORK(_work, _func) ... #define INIT_WORK(_work, _func) ... // //Specific workqueues operation to enqueue work on a work queue // #define create_workqueue(name) \ alloc_workqueue("%s", __WQ_LEGACY | WQ_MEM_RECLAIM, 1, (name)) static inline bool queue_work (struct workqueue_struct *wq, struct work_struct *work); static inline bool queue_delayed_work (struct workqueue_struct *wq, struct delayed_work *dwork, unsigned long delay); extern bool queue_work_on(int cpu, struct workqueue_struct *wq, struct work_struct *work); extern bool queue_delayed_work_on (int cpu, struct workqueue_struct *wq, struct delayed_work *work, unsigned long delay); extern void flush_workqueue (struct workqueue_struct *wq); extern void destroy_workqueue(struct workqueue_struct *wq); // //Kernel-global workqueues operation // static inline bool schedule_work (struct work_struct *work); static inline bool schedule_work_on(int cpu, struct work_struct *work); static inline bool schedule_delayed_work (struct delayed_work *dwork, unsigned long delay); static inline bool schedule_delayed_work_on (int cpu, struct delayed_work *dwork, unsigned long delay); static inline void flush_scheduled_work(void); // //Work operation // extern bool flush_work(struct work_struct *work); extern bool cancel_work_sync(struct work_struct *work); extern bool flush_delayed_work(struct delayed_work *dwork); extern bool cancel_delayed_work(struct delayed_work *dwork); extern bool cancel_delayed_work_sync(struct delayed_work *dwork); #define work_pending(work) \ test_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(work)) #define delayed_work_pending(w) \ work_pending(&(w)->work) ``` # Application ### How to use workqueues to implement a periodical timer? Below gist shows how to create a workqueue and register our callback function with recursive calls every 3 seconds to establish a periodical timer. {%gist ecfc640211c53a4e5fd23814b542a250 %} # Reference [^ref1]: https://www.ibm.com/developerworks/library/l-tasklets/index.html [^ref2]: https://elixir.bootlin.com/linux/v4.18.14/source/kernel/workqueue.c#L3056 https://elixir.bootlin.com/linux/v4.18.14/source/include/linux/workqueue.h [^ref3]: https://linuxtv.org/downloads/v4l-dvb-internals/device-drivers/ch01s06.html [^ref4]: http://www.hep.by/gnu/kernel/device-drivers/ch01s06.html [^ref5]: https://www.fsl.cs.sunysb.edu/kernel-api/ch01s05.html [^ref6]: http://www.embexperts.com/viewthread.php?tid=12&highlight=work%2Bqueue [^ref7]: http://nano-chicken.blogspot.com/2010/12/linux-modules73-work-queue.html