GPE Storm - HackMD

# GPE Storm  One may find his/her Linux machine always has 1 CPU under full load. In system monitor such as `top` or `htop`, a command called `kworker` takes up nearly 100% of a CPU. ## Observation This problem mostly happens on laptops. On my MSI GS60 6QD, with kernel `Linux 6.8.7-arch1-1 #1 SMP PREEMPT_DYNAMIC Wed, 17 Apr 2024 15:20:28 +0000 x86_64`, top two CPU usage listed in `top` are `kworker/0:1+kacpid` and `irq/9-acpi ` ``` PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 10 root 20 0 0 0 0 R 82.1 0.0 9:32.66 kworker/0:1+kacpid 79 root -51 0 0 0 0 R 15.6 0.0 1:48.09 irq/9-acpi ``` `kworker/0:1+kacpid` constantly takes high CPU usage, and `irq/9-acpi` constantly takes around 15% as well. This drains the battery fast and drives CPU fans to high load. It happens right from booting up. ## Investigation kworker refers to worker threads managed in worker-pool that consume work items in workqueue ([CMWQ](https://docs.kernel.org/core-api/workqueue.html)). Subsystems and drivers can create and queue work items through workqueue API functions as they see fit. The number after `kworker/` denotes the CPU core (in this case, core 0) and the specific worker id. These numbers are determined when `create_worker()` is called (defined in `kernel/workqueue.c`). `kacpid` stands for Kernel ACPI Daemon. [ACPI](https://en.wikipedia.org/wiki/ACPI) is a standard for handling power management and hardware configuration. The kacpid process is responsible for ACPI-related tasks such as managing power events and handling ACPI interrupts. `irq/9-acpi` indicates that it's an interrupt request associated with IRQ line 9 and ACPI. In [x86 IRQ](https://en.wikipedia.org/wiki/Interrupt_request#x86_IRQs), IRQ 9 is on Slave PIC, for ACPI interrupts on Intel Chipsets. The handling of interrupts has [two parts](https://0xax.gitbooks.io/linux-insides/content/Interrupts/linux-interrupts-9.html): top half and bottom half. Top half, which receives the hardware interrupt, needs to run as quickly as possible. Bottom half, where corresponding tasks to the interrupt are executed, is not as time-critical in non-real-time system and can be deferred for later execution. There are several options for this deferral: softirq, tasklet, workqueue and threaded interrupts. In this case, ACPI uses workqueue to do so. So far we can figure out that my computer is probably handling excessive numbers ACPI interrupts. We have two options: 1. Fix the hardware and firmware. Stop it from sending insane number of interrupts. 2. Stop handling the interrupts. ## Possible Solutions Stop handling the interrupts seems keep a fairly easy option. UEFI firmware on my laptop is closed-source software. Open-source ones such as Core Boot don't support my laptop, either. [This document](https://www.kernel.org/doc/Documentation/ABI/testing/sysfs-firmware-acpi) in kernel explains how to use [sysfs](https://en.wikipedia.org/wiki/Sysfs) to check ACPI firmware behavior, including interrupts. > However, one of the main functions of ACPI is to make the platform understand random hardware without special driver support. So while the SCI handles a few well known (fixed feature) interrupts sources, such as the power button, it can also handle a variable number of a "General Purpose Events" (GPE). > > A GPE vectors to a specified handler in AML, which can do a anything the BIOS writer wants from OS context. GPE 0x12, for example, would vector to a level or edge handler called \_L12 or \_E12. The handler may do its business and return. Or the handler may send send a Notify event to a Linux device driver registered on an ACPI device, such as a battery, or a processor. > > To figure out where all the SCI's are coming from, /sys/firmware/acpi/interrupts contains a file listing every possible source, and the count of how many times it has triggered > Besides this, user can also write specific strings to these files to enable/disable/clear ACPI interrupts in user space, which can be used to debug some ACPI interrupt storm issues. As suggested, use `grep . -r /sys/firmware/acpi/interrupts/` to list possible sources and triggered count: ```c error: 0 ff_gbl_lock: 0 disabled unmasked ff_pmtimer: 0 STS invalid unmasked ff_pwr_btn: 0 EN enabled unmasked ff_rt_clk: 0 disabled unmasked ff_slp_btn: 0 invalid unmasked gpe00: 0 invalid unmasked gpe0A: 0 invalid unmasked gpe0B: 0 invalid unmasked gpe0C: 0 EN enabled unmasked gpe0D: 0 invalid unmasked gpe0E: 0 invalid unmasked gpe0F: 0 invalid unmasked gpe01: 0 invalid unmasked ... ... ... gpe58: 0 invalid unmasked gpe59: 0 invalid unmasked gpe60: 0 invalid unmasked gpe61: 2461007 STS enabled unmasked gpe62: 0 EN enabled unmasked gpe63: 0 invalid unmasked gpe64: 0 invalid unmasked gpe65: 0 invalid unmasked gpe66: 9 EN enabled unmasked gpe67: 0 enabled unmasked gpe68: 0 invalid unmasked gpe69: 0 disabled unmasked gpe70: 0 invalid unmasked gpe71: 0 invalid unmasked gpe72: 0 invalid unmasked gpe73: 0 invalid unmasked gpe74: 0 invalid unmasked gpe75: 0 invalid unmasked gpe76: 0 invalid unmasked gpe77: 0 invalid unmasked gpe78: 0 invalid unmasked gpe79: 0 invalid unmasked gpe_all: 2461163 sci: 2461131 sci_not: 4 ``` As listed, GPE61 is triggered 2461007 times in only a few minutes. Disable GPE61 by `echo disable > /sys/firmware/acpi/interrupts/gpe61`. To disable it everytime system booted, add as a cron job in root user. One may think how about disable the interrupt right at [PIC](https://en.wikipedia.org/wiki/Programmable_interrupt_controller)? This may disable all the other normal ACPI interrupts. Along the above investigation, this could be the stepping stone of exploring interrupt mechanism in Linux kernel. ## Further investigation * How about masking it instead of disabling it? * Use `tools/workqueue/wq_monitor.py` or `tools/workqueue/wq_dump.py` to examine what GPE61 on my laptop really does. Find if disabling it poses any impact on the system. Not sure if my battery died pretty quick because of this. * [This post](https://askubuntu.com/questions/148726/what-is-an-acpi-gpe-storm) suggest "Transactions will use polling mode means that to handle the storm, the OS will stop using GPEs/interrupts to be informed of ACPI events and will instead -- on its own schedule -- "poll" or proactively ask the ACPI EC if any events it should know of have occurred. This way, the OS can still effectively perform ACPI functions while not being overwhelmed with a GPE storm." * Dig further into how interrupt works by following [this Lab](https://linux-kernel-labs.github.io/refs/heads/master/labs/interrupts.html). * A comment in a [LWN article](https://lwn.net/Articles/355700/) introducing Concurrency-managed workqueues: > Article: > ...The ACPI code had bound a workqueue thread to CPU 0 because some operations corrupt the system if run anywhere else; ... > > Comment: > (reply to someone being shocked) On some HPs, at least, certain ACPI operations trigger SMIs that then appear to be run on the CPU that triggered the SMI. HP's SMI handler seems to fail to restore CPU state if it runs on anything other than CPU 0. * This fits the observation that kworker works on CPU 0. * Notice that "disable interrupt" in the top half is different from this one we are doing here. Disabling interrupts in the top half is stop receiving all the other interrupts, which is a key latency concern for Linux to be hard-real-time.

Read more

Hey you say social media is broken but you do nothing about it

Audio Visual

2024q1 Homework5 (assessment)

2024q1 Homework1 (lab0)