--- tags: LINUX KERNEL, LKI --- # [Linux 核心設計](https://beta.hackfoldr.org/linux/): 中斷處理和現代架構考量 Copyright (**慣C**) 2019 [宅色夫](http://wiki.csie.ncku.edu.tw/User/jserv) ==[直播錄影 (上)](https://youtu.be/EjPWklezNrU)== ==[直播錄影 (下)](https://youtu.be/8ejnYR7A2Nc)== ## 簡介 中斷處理相信是人們不陌生的主題,甚至在中學生的計算機概論教材都出現這字眼,但在 Linux (或任何有規模的作業系統核心) 裡頭,中斷處理背後涉及的硬體特性、多種周邊 I/O、中斷控制器 (如是否支援 nested)、相關的排程和任務調度、延遲和即時處理等等,仍舊讓工程人員頭痛,特別將多核處理器、虛擬化技術,和為了實踐資訊安全而進行的隔離執行納入考量之後。 ![](https://i.imgur.com/YaxH1O1.png) 在本講座中,我們會從硬體特性著手,簡述 Intel 和 Arm 架構的中斷處理機制,再回頭看 Linux 核心的 softirq, tasklet, workqueue 用來實現中斷處理,並討論 request_threaded_irq / request_irq (這和引入 PREEMPT_RT 有高度關聯) 的用法,之後則針對多核處理器的架構去思考 Linux 核心的中斷處理做了哪些變革,搭配虛擬化和 Arm TrustZone 及 Intel SGX 的支援納入後,又有什麼需要深度分析的技術議題。 ## 中斷處理概念 - [ ] 取自 [COMS W4118: Operating Systems](http://www.cs.columbia.edu/~krj/os/index.html) 的 [Interrupts in Linux](http://www.cs.columbia.edu/~krj/os/lectures/L07-LinuxEvents.pdf) ![](https://i.imgur.com/csUJ42e.png) ![](https://i.imgur.com/WSmlKvr.png) * PIC translates IRQ to vector * Raises interrupt to CPU * Vector available in register * Waits for ack from CPU * Interrupts can have varying priorities * PIC also needs to prioritize multiple requests * Possible to “mask” (disable) interrupts at PIC or CPU ![](https://i.imgur.com/k606V2v.png) ![](https://i.imgur.com/EpXdwKK.png) ![](https://i.imgur.com/vmwkMIy.png) ![](https://i.imgur.com/MM2bjpJ.png) ![](https://i.imgur.com/zwyMIPL.png) ```clike /* <include/linux/interrupt.h> */ struct softirq_action { void (*action)(struct softirq_action *); }; ``` A kernel thread to process deferred softIRQs (per CPU) ```clike static void run_ksoftirqd(unsigned int cpu) { local_irq_disable(); if (local_softirq_pending()) { __do_softirq(); rcu_note_context_switch(cpu); local_irq_enable(); cond_resched(); return; } local_irq_enable(); } ``` ![](https://i.imgur.com/I4AZmAS.png) ![](https://i.imgur.com/gOqFJbm.png) * Creating a tasklet - DECLARE_TASKLET(name, func, data) * Scheduling tasklets - Tasklet function is runs every time softIRQ is executed - `tasklet_schedule( struct tasklet_struct *t)` - `tasklet_hi_schedule( struct tasklet_struct *t)` * Enabling/disabling tasklets - `tasklet_disable()` / `tasklet_kill()` - `tasklet_enable()` ![](https://i.imgur.com/vrvbze5.png) * Creating work queue - struct workqueue_struct *create_workqueue(char *name); * Creating work - DECLARE_WORK(name, void (*func)(void *), void *data); - INIT_WORK(struct work_struct *work, void (*func)(void *), void *data); * Scheduling work - int schedule_work(struct work_struct *work); - int flush_scheduled_work(void); - [ ] 取自 [CSE 438/598 Embedded Systems Programming](http://rts.lab.asu.edu/web_438_Fall_2014/CSE438_Fall2014_Main_page.htm) 的 [Linux Interrupt Processing and Kernel Thread](http://rts.lab.asu.edu/web_438/CSE438_598_slides_yhlee/438_7_Linux_ISR.pdf) ![](https://i.imgur.com/sZPlLU3.png) ![](https://i.imgur.com/Binka8K.png) * 為何有 ISR bottom half (`bh`) 呢?降低 interrupt latency,將工作切割為以下: - "top half", which receives the hardware interrupt and - "bottom half", which does the lengthy processing. * Top halves 的屬性和要求: - need to run as quickly as possible - run with some (or all) interrupt levels disabled - are often time-critical and they deal with HW - do not run in process context and cannot block ## 進行實驗 * 搭配 [I/O access and Interrupts](https://linux-kernel-labs.github.io/refs/heads/master/labs/interrupts.html) 實驗 * [Kernel 4.10 Examples of linux drivers](https://github.com/rrmhearts/linux-driver-examples) * 背景知識: [Linux Kernel Module Programming Guide](https://tldp.org/LDP/lkmpg/2.6/lkmpg.pdf) * 針對 Linux `v4.15` 需要做點調整: 將 `char/char_example.c` 的 `#include <asm/uaccess.h>` 換為 `#include <linux/uaccess.h>` (為了 `copy_to_user()`) * 測試 `char/`, `misc/`, `sysfs/bin_attr/` * Chapter 7 (Talking To Device Files, P.41), Chapter 12 (Interrupt Handlers, P.68) * [Char driver with interrupt handling](http://www.diegm.uniud.it/loghi/CE/slides/irq_driver.pdf) ## 加入 multi-core 和 threaded interrupt 的考量 - [ ] [Making Linux do Hard Real-time](https://www.slideshare.net/jserv/making-linux-do-hard-realtime) ![](https://i.imgur.com/yD7Nw4N.png) ![](https://i.imgur.com/bAPCaE6.png) * Threaded Interrupts Thread interrupts allows to use sleeping spinlocks * in PREEMPT_RT, all interrupt handlers are switched to threaded interrupt drivers in mainline get gradually converted to the new threaded interrupt API that has been merged in Linux 2.6.30. - [ ] [IRQs: the Hard, the Soft, the Threaded and the Preemptible](http://she-devel.com/Chaiken_ELCE2016.pdf) ![](https://i.imgur.com/hrNrOll.png) ![](https://i.imgur.com/1Dmtrpt.png) > Clock signals. `(a)` For level-triggered circuits. (b) For positive-edge triggering. `(c)` For negative-edge triggering. * [Interrupts and Interrupt Handling](https://0xax.gitbooks.io/linux-insides/Interrupts/) * O'Reilly [Interrupt Handling](https://www.oreilly.com/library/view/understanding-the-linux/0596005652/ch04s06.html) ## 虛擬化對中斷處理的影響 * [Linux 核心的中斷子系統](http://www.wowotech.net/irq_subsystem/interrupt_subsystem_architecture.html) * [I/O in Linux Hypervisors and Virtual Machines](https://github.com/fanjinfei/docs/blob/master/vmio_may9_2016.pdf) * [ARM Interrupt Virtualization](http://events17.linuxfoundation.org/sites/events/files/slides/ARM_Interrupt_Virtualization_Przywara.pdf) * GIC-400 是 GIC v2 架構,它是設計來配合 ARM CPU 的中斷控制器,能相容 TrustZone 和虛擬化架構、並最多支持到 8 個 ARM cores; * GIC-400 可以分成 4 大部分: Distributor (GICD_*), CPU interface (GICC_*), virtual control interface (GICH_*), virtual CPU interface (GICV_*),後面 2 個是為了虛擬化而存在; * [簡介 GIC-400](http://justinchunotes.blogspot.com/2018/12/gic-400.html)