Development Process

Overview

• Member training: Upstream Kernel Development (Part I)
• Member training: Upstream Kernel Development (Part II)
• Getting to Know the Linux Kernel: A Beginner's Guide - Kelsey Steele & Nischala Yelchuri, Microsoft
• Getting Into Linux Kernel Development After 30 Years - Muhammad Usama Anjum, Collabora
• Mentorship Session: Maintainership & Growing Influence in the Linux Kernel Community
• Beginner Linux kernel maintainers toolbox - Krzysztof Kozlowski

Kernel Release Model

• Kernel Recipes 2017 - Linux Kernel release model - Greg KH
• Day-to-day Testing of Linux 'Next' Kernel Branch - My Story - Marek Szyprowski, Samsung Electronics
• Kernel Recipes 2016 - From ‘git tag’ to the front page - Konstantin Ryabitsev
• An engineer's guide to Linux Kernel upgrades - Ignat Korchagin - NDC TechTown 2023

Coding Patterns

• Kernel Coding the Upstream Way - Tim Bird, Sony
• Kernel Recipes 2019 - Hunting and fixing bugs all over the Linux kernel
• Progress On Bounds Checking in C and the Linux Kernel - Kees Cook, Google & Gustavo A.R. Silva

Stable Releases

• Kernel Recipes 2017 - Linux Kernel release model - Greg KH
• Kernel Recipes 2015 - Linux Stable Release process - by Greg KH

git

Overview of git

• Git it right! [Workshop] - Git Merge 2022
• So You Think You Know Git - FOSDEM 2024
• So You Think You Know Git Part 2 - DevWorld 2024
• Advanced Git Magic - Pauline Vos - Full Stack Europe 2022
• CSESoc | Git Gud

Interrogators

• show, log, diff, blame etc.
• Backporting and conflict resolution
• So You Think You Know Git - FOSDEM 2024

git bisect

• Git driven debugging - Git Merge 2022
• Debugging with Git Bisect
• Using git bisect to pinpoint bugs in your commit history
• Howto: git bisect PPC64 PS3 Linux kernel USB regression!

git worktree

• How to use Git Worktree
• How to Use Git Worktree | Checkout Multiple Git Branches at Once

git rebase (interactive)

• Git Rebase Interactive :: A Practical Example
• Inserting a Commit into a Patch Series
• Reordering Commits in Git
• Squashing Git commits with Interactive Rebase

git rebase

• RubyConf 2019 - Story telling with Git rebase by Elle Meredith

git cherry-pick

• Backporting to Stable Branches

Internals of git

• The Git Parable - a different approach to understanding Git (90 min) - Johan Herland
• OSCON 2016: Dissecting Git's Guts - Git Internals - Emily Xie
• A tour of Git internals
• Git Internals - How Git Works - Fear Not The SHA!
• How does Git actually work? - CB Bailey & Andy Balaam [ACCU 2019]
• Deep Dive into Git - Edward Thomson
• Git Internals: a Database Perspective - Git Merge 2022

Getting Patches (from mailing lists)

Xapian Query Syntax

1. public-inbox help from any mailing list archive
2. Getting Started with Xapian
3. Xapian::QueryParser Syntax on the official documentation of Xapian

lei

1. Using lei, b4, and mutt to do kernel development
2. lore+lei: part 1, getting started
3. lore+lei: part 2, now with IMAP
4. lei-q(1)
5. Beginner Linux kernel maintainers toolbox - Krzysztof Kozlowski
6. Doing more with lore and b4 - Konstantin Ryabitsev (34:10 ~ FIN)

Applying Patches (from mailing lists)

1. git-am - Apply a series of patches from a mailbox
2. How to apply patches from the Linux Kernel Mailing List
3. Applying Patches To The Linux Kernel (obsolete)
4. am,shazam: retrieving and applying patches from B4 end-used doc.

Getting Patches (from git trees)

Backporting

• Backporting Linux Kernel patches
• Shung-Hsi Yu: Backporting BPF: Techniques and Challenges
• Backporting to Stable Branches
• Michal Kubeček: Backporting horror stories
• LAS16-101: Efficient kernel Backporting
• Contributing to the CentOS Stream Kernel - Prarit Bhargava and Don Zickus

Sending Patches

Upstreaming Process

• Submitting Your First Patch to the Linux Kernel and Responding to Feedback
• Getting to Know the Linux Kernel: A Beginner's Guide - Kelsey Steele & Nischala Yelchuri, Microsoft
• Baby Steps to Kernel Hacking - Sayli Yogesh Karnik, Credit Suisse
• Tutorial: From an Idea to a Patch in the Linux Mainline - Marta Rybczynska, Various Projects

b4 (Links)

1. B4 end-user documentation
2. Using lei, b4, and mutt to do kernel development
3. Use b4 for kernel contributions
4. Sending a kernel patch with b4 (part 1)
5. End-to-end patch attestation with patatt and b4

b4 (Videos)

• Doing more with lore and b4 - Konstantin Ryabitsev (00:00 ~ 34:10)
• Submit a kernel patch with b4
• Submit a follow-up patch with b4
• Beginner Linux kernel maintainers toolbox - Krzysztof Kozlowski (31:23)
• Comparing and Contrasting Patman Vs B4 for Posting Patches - Doug Anderson, Google (27:58)

patman

• Comparing and Contrasting Patman Vs B4 for Posting Patches - Doug Anderson, Google

Build System

Kconfig

• Introduction to Kernel Modules and Kbuild
• Managing Linux Kernel Configurations with Config Fragments - Darren Hart, VMware
• Krepair: Automatically Repairing .config Files to Cover Patches - Paul Gazzillo & Necip Yildiran

Kbuild

• A dive into kbuild - Cao jin, Fujitsu
• Introduction to Kernel Modules and Kbuild

Makefile

• Makefile Tutorial By Example
• Demystifying Linux Kernel Initcalls! - Mylène Josserand, Collabora (Example)

Building Kernel

• Linux bootcamp: From casual Linux user to kernel hacker - Part 3
• Tim Beale: Linux Kernel Development for Newbies
• https://javiercarrascocruz.github.io/kernel-build-tricks
• Embedded Linux from Scratch in 45 minutes, on RISC-V (Cross-compile example)

Kernel Image

• Linker scripts in the Linux Kernel, Kernel Image structure, and x86 boot sector debugging

Emulation

QEMU

• Fixing bugs in the Linux kernel with Syzbot, Qemu and GDB
• Finding your way through the QEMU parameter jungle
• Let's Build Our Own Virtual RaspberryPi Using QEMU Virtualization - Vipul Gupta, balena
• Qemu Demystified - Florian Wagner
• QEMU_4: System Mode Raspberry PI 3B

virtme-ng

• Kernel Recipes 2024 - virtme-ng
• Mentorship Session: Speeding Up Kernel Development With virtme-ng
• Speeding up Kernel Testing and Debugging with virt-meng - Andrea Righi

virsh

• Getting Started with Kernel-based Virtual Machine (KVM) - Leonard Sheng Sheng Lee, Computas

Testing

Testing overview

• "The magical fantasy land of Linux kernel testing" - Russell Currey (LCA 2020)
• Tips for Writing Good Tests for Linux - Tim Bird, Sony

Kselftest

• "How to write your first kernel selftest" - Michael Ellerman (LCA 2021 Online)
• Mentorship Session: Kernel Validation With Kselftest
• Kselftest Running in Test Rings - Where Are We? - Shuah Khan, The Linux Foundation
• YVR18-301: Kselftest framework roadmap and discussion

KUnit

• "KUnit: Past, Present, and Future" - David Gow (LCA 2021 Online)
• Mentorship Session: KUnit Testing Strategies
• XDC 2023 | October 17 | Status of KUnit tests on DRM subsystem | Arthur Grillo, Carlos Gallo Filho
• "Kernel Testing with KUnit: Bridging the Gap" - David Gow (LCA 2022 Online)

Linux Test Project (LTP)

• linux-test-project/ltp
• LinuxKernelTesting (playlist)

Existing Testing Infrastructures

Linux Kernel Functional Testing (LKFT)

• Linux Kernel Functional Testing

KernelCI

• ELISA Seminar (September 2024) – Meet the New KernelCI

Syzbot Board

• syzkaller.appspot.com

Debugging Overview

Kernel Debugging Overview

• Tools and Techniques to Debug an Embedded Linux System - Sergio Prado, Embedded Labworks
• Tools and Techniques to Debug an Embedded Linux System
• Mentorship Session: Linux Kernel Debugging Tricks of the Trade
• Feb 8 at 5pm GMT "Kernel Debug Stories for Arm" (in English)

Print Debugging

Dynamic Debug

• dynamic-debug-howto
• How to use the kernel dynamic debug
• SAN19-424 Event Tracing and Pstore with a pinch of Dynamic debug (37:01)

Debugfs

• DebugFS from kernel doc

OOPS and Post-Mortem

Kernel OOPS

• If (oops) { Do_not_panic(); } - Lucky Tyagi, Samsung Semiconductor India R&D Center
• The tale of a single register value
  
• Tools and Techniques to Debug an Embedded Linux System
• admin-guide/bug-hunting.html

decode_stacktrace.sh

• Decoding stack traces in the Linux kernel
• The tale of a single register value
• Mentorship Session: Dynamic Program Analysis for Fun and Profit (31:39)

pstore

• Recovering the Linux kernel logs from a previous boot with pstore and ramoops
• SAN19-424 Event Tracing and Pstore with a pinch of Dynamic debug

Kdump

• Kdump in an hour
• Postmortem Analysis with Kexec/kdump and Ftrace - Steven Rostedt, Google
• Kernel Recipes 2015 - Kernel dump analysis - by Adrien Mahieux
• Marian Marinov - Analyzing Linux kernel crash dumps
• Kernel dump analysis made quick and easy!

Kernel Tracing

Dynamic Tracing Overview

• Steven Rostedt on the Linux Kernel's "Hello World"
• Mentorship Session: An Introduction to Linux Tracing and its Concepts
• Tracing Summit 2022 - State of the Linux Tracers

ftrace (trace-cmd)

• See what your computer is doing with Ftrace utilities
• Finding Sources of Latency on your Linux System - Steven Rostedt, VMware
• Embedded Recipes 2022 - Tracing on embedded boards

ftrace (tracefs)

• Steven Rostedt - Learning the Linux Kernel with tracing
• BKK19-TR04 - Fantastic tracepoints and where to find them
• LF Live Mentorship Session: Tracing with Ftrace: Critical Tooling for Linux Development
• Embedded Fest 2021. Steven Rostedt. Dynamic tracing within a busybox environment

LTTng

• Tracing Summit 2022 - LTTng: Beyond Ring-Buffer Based Tracing
• [LEE3] System wide tracing with LTTNG Philip Dylan Gleonec, Savoir Faire Linux
• Fine-grained Distributed Application Monitoring Using LTTng

Event Sources

Events in the Kernel

• Probes in the kernel, where it goes - Mr HIRAMATSU, Masami
• Mentorship Session: An Introduction to Linux Tracing and its Concepts
• pt_regs - the good the bad and the ugly - Florent Revest

Tracepoints

• LF Live Mentorship Session: Tracing with Ftrace: Critical Tooling for Linux Development (07:03)
• BKK19-TR04 - Fantastic tracepoints and where to find them
• DEF CON 30 - Rex Guo, Junyuan Zeng - Trace me if you can: Bypassing Linux Syscall Tracing
• SAN19-424 Event Tracing and Pstore with a pinch of Dynamic debug

Kprobes

• Where have all the kprobes gone - Jiri Olsa
• Kprobes internals - Thomas Bitzberger - LSE Week 2017 (FR)

Probe-Hooking APIs in the kernel

• Jiri Wiesner Writing debugging modules

BPF

bpftrace

• Linux tracing made simpler with bpftrace - DevConf.CZ 2022
• An introduction to bpftrace tracing language - DevConf.CZ 2020
• bpftrace internals - DevConf.CZ 2020

BCC

• Jan Kara Tracing and eBPF hands on session
• Linux Performance with eBPF and bcc-tools
• Tutorial: Getting Started with eBPF - Liz Rice, Isovalent
• Linux Tracing with BPF, BCC and More - Mauricio Vásquez Bernal & Alban Crequy, Kinvolk

Debuggers

KGDB

• Debugging the Linux kernel with GDB
• Debugging the Linux kernel with GDB (Video)
• KGDB (playlist)
• Using Serial kdb / kgdb to Debug the Linux Kernel - Douglas Anderson, Google

drgn

• Michal Koutný Meeting drgn
• Kernel Recipes 2022 - Powerful and Programmable Kernel Debugging with drgn

Compiler Instrumentation (Saniziters)

Sanitizer Overview

• Kernel Sanitizers Office Hours - POTAPENKO Alexander, VYUKOV Dmitry, COOK Kees, ELVER Marco, MCKE…
• Mentorship Session: Dynamic Program Analysis for Fun and Profit
• ELISA Summit (Sep 2022)| Using Memory Access Error Detection for Safety Argumentation
• google/kernel-sanitizers

KASAN

• Embedded Fest 2021. Ruslan Bilovol. Linux Kernel hardening techniques
• Sanitizing the Linux Kernel — On KASAN and other Dynamic Bug-finding Tools - Andrey Konovalov
• Mitigating Linux kernel memory corruptions with Arm Memory Tagging | Linux Security Summit 2021
• Jerome Marchand: KASan or how to get rid of memory corruption

KMEMLEAK

• Embedded Fest 2021. Ruslan Bilovol. Linux Kernel hardening techniques

KFENCE

• ELISA Summit (Sep 2022)| Using Memory Access Error Detection for Safety Argumentation
• google/kernel-sanitizers
• Linux SLUB Allocator Internals and Debugging - KFENCE, Part 4 of 4

UBSAN

• Embedded Fest 2021. Ruslan Bilovol. Linux Kernel hardening techniques

KCSAN

• Mentorship Session: The Kernel Concurrency Sanitizer
• LPC 2020 - Refereed Track - Day 1 (01:59:00) (title: Data-race detection in the Linux Kernel)
• Concurrency bugs should fear the big bad data-race detector (part 1)
• Concurrency bugs should fear the big bad data-race detector (part 2)

Lockdep

• What Lockdep Does and Doesn't - Byungchul Park, LG Electronics
• LPC2018 - Recursive read deadlocks and Where to find them

Static Tools

sparse

• Mentorship Session: Static Analysis & Tools
• Sparse warnings

smatch

• Mentorship Session: New Ideas for Smatch (Static Analysis)
• MAD24 216 What Can Static Analysis Do For You Smatch
• Linux Kernel Development - Smatch

Coccinelle

• Mentorship Session: Coccinelle: Automating Large-scale Evolution and Bug Finding in C Code
• SAN19-500K1 Keynote: Coccinelle: 10 Years of Automated Evolution in the Linux Kernel

Sources of Concurrency

• Any PREEMPT_RT talk: figure out what was changed by PREEMPT_RT – and revert it back!
• Unreliable Guide To Hacking The Linux Kernel
• Hint: you can use latency output of trace-cmd to know the context (IRQ, softirq, process) a function is in.
• See the description in -l option of trace-cmd-report(1).
• Also see Latency trace format.

IRQ

• A Journey into Linux Interrupt Handling Bottom Halves - Allen Pais, Microsoft
• IRQs: the Hard, the Soft, the Threaded and the Preemptible
• Understanding Linux Interrupt Subsystem - Priya Dixit, Samsung Semiconductor India Research

softirq

• A Journey into Linux Interrupt Handling Bottom Halves - Allen Pais, Microsoft
• Getting Around to It: Deferred Work in the Linux Kernel - Alison Chaiken, Aurora Innovation
• IRQs: the Hard, the Soft, the Threaded and the Preemptible
• Unblocking the softirq lock for PREEMPT_RT - Sebastian Siewior

Processes

• Context Switching in the Linux Kernel - Sharad
• Tutorial: Leveraging the OS CPU Scheduler to Write Real-Time MT Apps - Kaiwan Billimoria, kaiwanTECH
• Thread implementation #5 - Linux context switch | cs370
• Breaking and Protecting Linux Kernel Stack - Elena Reshetova, Intel
• Threads in Embedded Linux- 6 Easy Pieces - Loïc Domaigné, Doulos
• The Demystifying the Linux CPU Scheduler book by sysprog21 (not yet published).

Preemption

• (Part 2, 9am GMT) Using Linux-Kernel for Real-Time systems (51:15)
• Giovanni Gherdovich: Recent works on kernel preemption models
• Giovanni Gherdovich The four kernel preemption flavors in Linux
• Real time Linux - Bogdan Lezhepekov, Radoslav Kolev (18:15)
• An introduction to real-time Linux
• Kernel Recipes 2023 - Coming soon

Deferring Workload (to processes)

Workqueue

• A Journey into Linux Interrupt Handling Bottom Halves - Allen Pais, Microsoft
• Getting Around to It: Deferred Work in the Linux Kernel - Alison Chaiken, Aurora Innovation (24:02)
• What You Always Wanted to Know About Workqueues

Threaded IRQ

• What Every Driver Developer Should Know about RT - Julia Cartwright, National Instruments (08:55)
• TUT 1279 The four kernel preemption flavors in Linux (17:02)

kthread

• Let's code a Linux Driver - 14: Threads in a Linux Kernel Module

Deferring Workload (to softirq)

tasklet

• The end of tasklets on LWN
• Modernizing the tasklet API on LWN
• IRQs: the Hard, the Soft, the Threaded and the Preemptible (37:02 ~ 39:59)

Schedulers

Scheduler Overview

• (Part 1, 9am GMT) Using Linux kernel for Real-Time Systems (21:49)
• (Part 2, 9am GMT) Using Linux-Kernel for Real-Time systems (12:08)
• Brief History of Linux CPU Scheduler - Huichun Feng
• YVR18-220:The Linux Kernel Scheduler (For Beginners)
• BKK19-TR03 - The Linux Kernel Scheduler - Overview
• SAN19-220 Deep dive in the scheduler

fair_sched

• (Part 1, 9am GMT) Using Linux kernel for Real-Time Systems (21:49)
• (Part 2, 9am GMT) Using Linux-Kernel for Real-Time systems (12:08)
• Michal Koutný: EEVDF is the new (sched_)normal
• From Linux CFS Scheduler to Goroutine Scheduler
• SAN19-419 Why you should use the SCHED_IDLE CFS scheduling policy
• Introduction to SCHED_NORMAL (Peter Zijlstra)

rt_sched

• (Part 1, 9am GMT) Using Linux kernel for Real-Time Systems (32:19)
• (Part 2, 9am GMT) Using Linux-Kernel for Real-Time systems (20:22)

dl_sched

• (Part 1, 9am GMT) Using Linux kernel for Real-Time Systems (38:10)
• (Part 2, 9am GMT) Using Linux-Kernel for Real-Time systems (23:08)
• SCHED_DEADLINE semi partitioned scheduler and dlmon
• Embedded Recipes 2017 - Understanding SCHED_DEADLINE - Steven Rostedt
• Results from using SCHED_DEADLINE for energy-aware optimization of RT DAGs on heterogeneous hardware

Locks

spinlock

• Real time Linux - Bogdan Lezhepekov, Radoslav Kolev (20:44)
• Alex Kogan — New locks for the old kernel
• Parallelisation in the Linux Kernel - Handmade Seattle 2020
• Also see Unreliable Guide To Locking on kernel doc.

mutex

• Let's code a Linux Driver - 28: Mutex exclusive access to shared resources
• A sleeping lock. Unable to protect against things that are out of scheluer's control (e.g. IRQ and softirq).

semaphore

• The shrinking role of semaphores on LWN

RCU

• Decoding Those Inscrutable RCU CPU Stall Warnings
• Mentorship Session: Unraveling RCU-Usage Mysteries (Fundamentals)
• Mentorship Session: Unraveling RCU-Usage Mysteries (Additional Use Cases)

Barriers

Compiler Barrier

• Mentorship Session: The Kernel Concurrency Sanitizer
• Status Report: Broken Dependency Orderings in the Linux Kernel - Marco Elver, Paul Heidekrüger

CPU Barriers

• Mentorship Session: The Kernel Concurrency Sanitizer
• Does an x86 CPU Reorder Instructions?
• Arm's Weakly-Ordered Memory Model and Barrier Requirements - Ash Wilding, Amazon

Memory Model

• Mentorship Session: The Kernel Concurrency Sanitizer
• More Science Fiction than Fairy Tale: A Discussion of the Linux Kernel Memory Mo… Jonas Oberhauser
• P99 CONF 2023 | How to Avoid Learning the Linux-Kernel Memory Model by Paul McKenney
• Kernel Recipes 2024 - Case Study: Concurrent Counting

System Calls

System Calls

• Syscalls and Strace
• What Have Syscalls Done for you Lately?
• MAD24 315 Rethinking the kernel system call entry
• Linux ABI - Syscalls in Assembly and Calling Assembly from C
• For x86-64 syscall/sysret instruction, see videos in Arch2001 and its full playlist.

strace

• Strace: Monitoring The Kernel-User-Space Conversation - Michael Kerrisk
• Modern strace - DevConf.CZ 2023
• Tutorial: Debugging with Strace - A Peek Behind the Scenes of Linux Processes - Avikam Rozenfeld
• Using strace to troubleshoot issues - DevConf.CZ 2021
• Using Strace to Understand GitLab - Part 1: Intro
• Using Strace to Understand GitLab - Part 2: Git

vDSO

• Using GNU poke: looking at the vDSO

Timers and Timekeeping

Timers and Timekeeping

• Mentorship Session: The Ticking Beast: a Deep Dive Into Timers, Timekeeping, Tick & Tickless Kernels
• Timekeeping in the Linux Kernel - Stephen Boyd, Qualcomm Innovation Center

hrtimer

• Let's code a Linux Driver - 9: High Resolution Timer in a Linux Kernel Module

Memory Management Overview

Overview

• Tutorial: Linux Memory Management and Containers - Gerlof Langeveld, AT Computing
• Inspecting and Optimizing Memory Usage in Linux - João Marcos Costa, Bootlin
• Kernel Recipes 2024 - All your memory are belong to… whom?
• Introduction to Memory Management in Linux
• Page table reclaim - David Hildenbrand
• Introduction to Memory Management in Linux by Alan Ott_HD.mp4

Virtual Address Spaces

Address Space Layout (Talks)

• Tutorial: Linux Memory Management and Containers - Gerlof Langeveld, AT Computing
• LVC20 106 4G 4G virtual memory split in arm32 kernel
• Introduction to Memory Management in Linux
• Introduction to Memory Management in Linux by Alan Ott_HD.mp4
• February 20th Linaro Forum for Arm Linux Kernel Topics

Address Space Layout (Links)

Division of virtual memory in arm32:

1. https://people.kernel.org/linusw/how-the-arm32-kernel-starts
2. https://people.kernel.org/linusw/setting-up-the-arm32-architecture-part-1
3. https://people.kernel.org/linusw/setting-up-the-arm32-architecture-part-2

For the x86_64:

1. https://www.kernel.org/doc/Documentation/x86/x86_64/mm.txt

The Kernel Space

• Tutorial: Linux Memory Management and Containers - Gerlof Langeveld, AT Computing
• Direct Map Management - Mike Rapoport

Physical Address Space

Physical Address Space

= Mentorship Session: Debugging Linux Memory Management Subsystem

• Arch4001 03 Input Output 01 MMIO 01 Overview
• Arch4001 03 Input Output 01 MMIO 02 What Does It Mean To Be Memory Mapped
• SUSE Labs Conference 2020 - DMA mapping for the Raspberry Pi 4

Pages

• Tutorial: Linux Memory Management and Containers - Gerlof Langeveld, AT Computing (27:34)
• Mentorship Session: Debugging Linux Memory Management Subsystem
• Inspecting and Optimizing Memory Usage in Linux - João Marcos Costa, Bootlin (3:31)
• Kernel Recipes 2024 - All your memory are belong to… whom?
• Linux Memory Management - VMScan | DEEP LINUX

Zones (Links)

1. Physical Memory in the Linux Kernel Documentation.
2. Hierarchical NUMA on LPC2017 and its audio recording.
3. How the Linux kernel divides up your RAM in Chris's Wiki.

Zones (Talks)

• Tutorial: Linux Memory Management and Containers - Gerlof Langeveld, AT Computing (43:23)
• Inspecting and Optimizing Memory Usage in Linux - João Marcos Costa, Bootlin (5:42)
• SUSE Labs Conference 2020 - DMA mapping for the Raspberry Pi 4 (27:25)

Memory Allocation in Kernel

Page Allocator

1. Linternals: Memory Allocators [0x01]
2. 13. Memory Management in Linux Kernel

Slab Allocator

• Kernel used to have three flavors of slab allocators: SLOB, SLAB, and SLUB.
• SLOB and SLAB were removed. SLUB is the only one remaining.
• SLUB Internals for Exploit Developers - Andrey Konovalov, xairy.io
• And its slides are also worth looking into.
• ECE-T480 - Spring 2021: Lecture 16 (the slab allocator)
• How to display kernel slab cache information using Slabtop

kmalloc()

• Memory Allocation Guide from kernel doc

vmalloc()

• Memory Allocation Guide from kernel doc

Paging (Hardware)

x86-64

1. Paging part of the Architecture 2001: Intel x86-64 OS Internals.
2. Mentorship Session: Debugging Linux Memory Management Subsystem

arm64

• "Large Virtual Address support (52-bit) in ARM64 kernel" - Bhupesh Sharma (LCA 2021 Online)
• LVC21-221 Large Virtual Address support 52 bit in ARM64 kernel

RISC-V (Links)

1. The RISC-V Instruction Set Manual, Volume II: Privileged Architecture.
2. 4.3 Sv32: Page-Based 32-bit Virtual-Memory Systems
3. 4.4 Sv39: Page-Based 39-bit Virtual-Memory System
4. 4.5 Sv48: Page-Based 48-bit Virtual-Memory System
5. 4.6 Sv57: Page-Based 57-bit Virtual-Memory System
6. RISC-V: mm: Make SV48 the default address space in LWN

RISC-V (Videos)

• COSC562 - Sv39 MMU System

The Userspace

vm_area_struct

• Process Address Space: The way to create virtual address (page table) of userspace application by Adrian Huang
• ProcessAddressSpace
• Tracing on Page Table - YuHsiang Tseng & ChinEn Lin, National Taiwan Ocean University
• Lower Response Time of Fork by Extending Copy-on-write to the Page Table - Chih-En Lin
• Structures to manage the stack, virtual memory, and process ids in the Linux kernel
• Hiding Process Memory via Anti-Forensic Techniques
• The Maple Tree + Lock Scalability - Liam Howlett and Michel Lespinasse

Paging

Page Tables

• ELISA Seminar (June 2022): What is a page table?
• Mentorship Session: Huge Page Concepts in Linux (first 30 minutes or so)

Page Faults

• Kernel Recipes 2023 - Faster & Fewer Page Faults
• The Maple Tree + Lock Scalability - Liam Howlett and Michel Lespinasse
• Long Live Asynchronous Page Fault! - Vitaly Kuznetsov & Vivek Goyal, Red Hat

Page Reclam

• Overview of memory reclaim in the current upstream kernel - Vlastimil Babka
• MGLRU - Yu Zhao
• Page table reclaim - David Hildenbrand

Other Topics

History of Slab Allocators in Kernel

• Kernel Exploitation - Slab Allocators
• Vlastimil Babka The rise and fall of kernel slab allocators
• Reducing the Kernel's Slab Allocators: Progress Report - Vlastimil Babka, SUSE
• [PATCH v2 0/6] remove SLOB and allow kfree() with kmem_cache_alloc()
• [PATCH v2 00/21] remove the SLAB allocator

Driver Core

Kernel Modules

• 2 ELC-2018 EALE Introduction to Kernel Modules - Behan Webster
• Linux bootcamp: From casual Linux user to kernel hacker - Part 4
• Understanding the Structure of a Linux Kernel Device Driver
• Into the Jungle - How to write a Linux kernel driver for any subsystem - Marta Rybczynska

Driver Model

• Embedded Recipes 2019: Herd your socs become a matchmaker
• Linux Matchmaking: Helping devices and drivers find each other
• Embedded Linux kernel and driver development training by Bootlin (free access).
• 5 ELC-2018 EALE I2C Drivers - Michael Opdenacker: example using I2C bus.
• Working with devicetree: Free, but can only be accessed after registration.

udev

• Introducing Udev on Oracle Linux
• [ENG] Christian Brauner: "Making the Kernel and Udev Namespace Aware" / #LinuxPiter
• Dynamic kernel device management with udev
• An introduction to Udev: The Linux subsystem for managing device events
• udev: Device Manager for the Linux Kernel in Userspace
• udev: Function Flow for KOBJECT_UEVENT kernel group message

Hardware Descriptions

Device tree

• Webinar On-Demand: Demystifying Device Tree for NXP® i.MX Processors
• Enabling new hardware on embedded Linux (from schematics to the device tree)
• Demystifying Device Tree Concepts - Priya Dixit
• FOSDEM 2024 - From phone hardware to mobile Linux
• Device Tree: hardware description for everybody !
• 5 ELC-2018 EALE I2C Drivers - Michael Opdenacker: example using I2C bus.

ACPI

• ACPI Source Language (ASL) Tutorial from Intel.
• Andy Shevchenko | ACPI overlays for embedded devices
• Kernel Recipes 2015 - Representing device-tree peripherals in ACPI - by David Woodhouse

Buses (that don't enumerate)

(There are more buses in the world, but those are the more easily accessible ones.)

Platform

• BoF: Early Platform Drivers in Linux Kernel - Bartosz Golaszewski, BayLibre (first 5 minutes)

I2C

• Basics of I2C on Linux - Luca Ceresoli, Bootlin
• 5 ELC-2018 EALE I2C Drivers - Michael Opdenacker
• Tutorial: Introduction to I2C and SPI: Both In-kernel and In-userspace - Michael Welling
• Custom I2C peripheral in QEMU

SPI

• 6 ELC-2018 EALE SPI Drivers - Michael Welling
• Tutorial: Introduction to I2C and SPI: Both In-kernel and In-userspace - Michael Welling
• Groking the Linux SPI Subsystem - Matt Porter, Konsulko
• Custom SPI peripheral in QEMU

Buses (that enumerate)

PCI and PCIe

• Some of the videos in Arch4001 (playlist)
• Linux PCI(e) Driver Tutorial (playlist)

USB

• A Tour of USB Device Controller (UDC) in Linux - Hervé Codina, Bootlin
• Debugging Usually Slightly Broken (USB) Devices and Drivers - Krzysztof Opasiak, Samsung
• Watch Linux kernel developer write a USB driver from scratch in just 3h for Apple Xserve front-panel

Resource Abstraction

GPIO

• Introduction to Pin Muxing and GPIO Control Under Linux - Neil Armstrong, BayLibre
• Linus Walleij - GPIO and Pin Control for Embedded Systems
• GPIO for Engineers and Makers
• Linux GPIO: Evolution and Current State of the User API
• [ENG] Bartosz Golaszewski: "New GPIO interface for linux user space"

IRQ (General)

• IRQs: the Hard, the Soft, the Threaded and the Preemptible
• Getting Around to It: Deferred Work in the Linux Kernel - Alison Chaiken, Aurora Innovation
• Understanding Linux Interrupt Subsystem - Priya Dixit, Samsung Semiconductor India Research

IRQ (x86)

• [x86] Linux Kernel Interrupt Delivery Configuration: How the Linux Kernel Interacts with Hardware - Adrian Huang, Lenovo

IRQ (Arm)

• Asahi Linux: PCI power management
• How Dealing with Modern Interrupt Architectures can Affect Your Sanity
• LVC20 102 Interrupt sub system in ARM boards using Xilinx Zynq Board
• Asahi Linux: PCI power management (first hour or so)

Pinctrl

• Introduction to Pin Muxing and GPIO Control Under Linux - Neil Armstrong, BayLibr
• Tutorial: Spelunking for Hardware Data - Matt Porter, Konsulko Group
• Linus Walleij - GPIO and Pin Control for Embedded Systems

regmap

• Simplify and Reuse Your Driver's Code with Regmaps - Ioan Adrian Ratiu, Collabora Ltd

clocks

• Mentorship Session: The Ticking Beast: a Deep Dive Into Timers, Timekeeping, Tick & Tickless Kernels
• Timekeeping in the Linux Kernel - Stephen Boyd, Qualcomm Innovation Center
• Tutorial: What The Clock! - Linux Clock Subsystem Internals - Neil Armstrong, BayLibre SAS
• Updating and Modernizing Clk (Clock) Drivers - Chen-Yu Tsai, Google

DMA

• Taming DMA: Tales Wrestling Memory Corruption - Ahmad Fatoum, Pengutronix
• Dmaengine overview on STM32MP WiKi
• Laurent Pinchart - mastering the dma and iommu apis | ELC 2014
• Linux DMA In Device Drivers
• Using DMA in a Linux PCI or PCI Express Driver

Other topics

DT Schema

• How to Get Your DT Schema Bindings Accepted in Less Than 10 Iterations - Krzysztof Kozlowski, Linaro

Firmware APIs

• The linux-firmware repository
• Unravelling How Firmware Interacts with the Linux Subsystem - Smitha T Murthy
• Getting a Time of Flight Camera Working in Linux, the Full Story from Kernel to User Space
• Linux Kernel Development - Firmware API

device_link

• Kernel Recipes 2019 - What To Do When Your Device Depends on Another One

Overview

Overview

• Kernel Recipes 2015 - Introduction to Kernel Power Management - by Kevin Hilman
• Power management on Linux: from the hardware to the kernel and userspace interfaces
• Power management techniques on embedded Linux
• YVR18-221:Linux Kernel Power Management Features - An Overview
• BKK19-TR02 - Linux Kernel Power Management - 101
• Michael Turquette: "Introduction to Power Management" [ENG]

System-wide Suspend

System Sleep States (Talks)

• Is Linux Suspend ready for the next decade - Len Brown
• Evolution of Suspend-to-Idle Support in The Linux Kernel - Rafael Wysocki (LCA 2021 Online)
• BKK19-119 - Device power management and idle

System Sleep States (Links)

1. System Sleep States of the kernel documentation.
2. Power management/Suspend and hibernate in Arch Wiki
3. Suspend and hibernate in Gentoo Wiki.
4. include/linux/pm.h in Linux kernel.
5. Device Power Management Basics in Linux Kernel Documentaion to see how devices handle the system-wide suspension.

Freeze (s2idle)

• Evolution of Suspend-to-Idle Support in The Linux Kernel - Rafael Wysocki (LCA 2021 Online)

Standby

• Is Linux Suspend ready for the next decade - Len Brown (02:31) : "…standby is sort of deprecated. I haven't seen standby on a machine for years."

Deep (s2ram)

• Suspend-to-RAM
• Unlocking the Potential of Suspend to RAM Using Linux in a Multi-Core, Multi-Firmware Automotive SoC - Grégory Clement, Bootlin

Runtime PM, devices

Runtime PM

• Adding Runtime Power Management Capabilities to Device Drivers - Shreeya Patel, Collabora
• SAN19-421 Training: Device power management for idle
• BKK19-119 - Device power management and idle
• LPC2019 - Integration of PM-runtime with System-wide Power Management
• The Fall of the Legacy - Vaibhav Gupta, Open Source Contributor

PM Domain (genpd)

• Kernel Recipes 2017 - Overview of Generic PM Domains (genpd) - Kevin Hilman
• The Case for an SoC Power Management Driver - Stephen Boyd, Google
• Adding Runtime Power Management Capabilities to Device Drivers - Shreeya Patel, Collabora
• SAN19-421 Training: Device power management for idle
• Active state management of power domains

Device Link

• Kernel Recipes 2019 - What To Do When Your Device Depends on Another One

DVFS

• LLC 2022 - Interconnect and Devfreq on i.MX - Bus Dynamic Frequency Scaling, by Abel Vesa

CPU Runtime

cpufreq

• SUSE Labs Conference 2020 - The schedutil frequency governor: current status and performance
• Kernel modification: Adding Governors
• Advances in CPU Frequency Management
• LinuxDays 2018 - Frequency scaling governor for the Linux kernel - Giovanni Gherdovich
• LinuxDays 2017 - CPU Frequency Scaling in Linux - Giovanni Gherdovich

OPP

• Operating Performance Points (OPP) Library in Linux Kernel Documentation.
• Power management techniques on embedded Linux (21:50)

CPUIdle

CPUIdle

• [ENG] Rafael J. Wysocki: "Advances in CPU Idle Time Management in Linux" / #LinuxPiter

ACPI Power States

S States

• Arch4001 07 Power Transition 01 Intro 01 Intr
• Computer Architecture: CPU ACPI Power Management lecture 1
• Improving monitoring of power saving states - Mr Stanislaw Kardach (Google), Sven van Ashbrook

P States

• 07 - OSPM 2024 - Rafael J. Wysocki, Intel - Energy Models and Intel chips
• A New CPU Frequency Control Mechanism on Linux - Ray Huang, AMD
• Power Capping Linux - Srinivas Pandruvada, Intel

C States

1. CPU idle power saving methods for real-time workloads on Linux Foundation Wiki
2. Excerpt of Energy Efficient Servers: Blueprints for Data Center Optimization
3. Tuning Linux for Performance - I Wanna Go Fast! - Anthony Nocentino - PSConfEU 2023
4. Tuning RT Kernel to Improve Schedule Performance with Intel Platform - Junxiao Chang, Intel

S0ix

1. The S0ixSelftestTool
2. PCH S0 Low Power
3. Power Management Sub-state
4. Prepare hardware for modern standby on the Windows documentation.
5. Descriptions on the ChromeOS documentation
6. Host and Guest Suspend Under Xen - S3 and S0ix - Marek Marczykowski-Górecki, Invisible Things Lab

Other Frameworks

PM QoS

• CPU Power Saving Methods for Real-time Workloads - Ramesh Thomas, Intel

Thermal Management

Thermal Subsystem

• Improvisation and Demonstration of Linux Thermal Framework for Multiple Temperature Sensors
• LVC20 110 Skin temperature management with hierarchical constraints

thermald

• thermald

Display Interfaces

VGA

• M14 - 3 - Basic Operation of a CRT Monitor
• M14 - 4 - VGA - Horizontal Synchronization
• M14 - 5 - VGA - Vertical Synchronization
• M14 - 6 - VGA - Pixel Clock Rate
• Nandland Go Board Project 9 - Introduction to VGA

RGB

• pico-rgb2hmi - video converter - Marcelo Lorenzati - Commodore Users Europe - 17th of Sep. 2022
• Image interfaces in LCD modules explanation - LVDS, RGB, MIPI, Vx1, eDP interfaces explanation

LVDS

• LVDS Display Panel
• Video signal transmission between motherboard and Tcon via LVDS. VESA and JEIDA standard
• Testing and Remote Access to Embedded System DPI/LVDS Display Output - Marek Vasut
• What is LVDS … Old laptop Screen reuse

DPI

• Testing and Remote Access to Embedded System DPI/LVDS Display Output - Marek Vasut

HDMI

• Display support on Embedded Systems: Linux implementation & limitations (Neil Armstrong, Linaro)
• HDMI 4k Video: Lessons Learned - Hans Verkuil, Cisco Systems Norway
• HDMI CEC: What? Why? How?

DisplayPort

• What is DisplayPort (DP)?
• TI Precision Labs - Video Interface: Understanding DP Link Training
• Essentials of DisplayPort Protocols at HBR3 8.1 Gb/s Link Rates
• Display support on Embedded Systems: Linux implementation & limitations (Neil Armstrong, Linaro)
• A Journey through Upstream Atomic KMS to Achieve DP Compliance - Manasi Navare, Intel

DP MST

• XDC 2022 | Introduction to DisplayPort MST support on Linux | Lyude Paul
• XDC 2022 Demo | Progress of MST improvement from AMD | Wayne Lin

EDID

• Extended Display Identification Data (EDID): Problems, Pitfalls and Complications - Hans Verkuil

KMS

Display Pipeline

• ELCE 2022: Navigating the Linux Graphics Stack
• Walking Through the Linux-Based Graphics Stack - Paul Kocialkowski, Bootlin
• Graphics: A Frame's Journey | FOSDEM 2023
• An introduction to the Linux graphics stack
• The Linux graphics stack in a nutshell, part 1
• The Linux graphics stack in a nutshell, part 2
• DRM KMS overview

KMS (Recent)

• Getting pixels on screen on Linux: introduction to Kernel Mode Setting - Simon Ser
• A Current Overview of the DRM KMS Driver-Side APIs - Paul Kocialkowski, Bootlin
• ELCE 2022: Navigating the Linux Graphics Stack
• Graphics: A Frame's Journey | FOSDEM 2023

KMS (Old, but good!)

• Kernel Recipes 2017 - An introduction to the Linux DRM subsystem - Maxime Ripard
• Kernel Recipes 2015 - Anatomy of an Atomic KMS Driver - by Laurent Pinchart
• XDC 2019 | A case study on frame presentation from user space via KMS - Heinrich Fink
• LVC21F-108 Advanced KMS
• Demystifying Linux MIPI DSI Subsystem - Jagan Teki, Amarula Solutions
• Embedded Linux Conference 2013 - Anatomy of an Embedded KMS Driver
• Linux DRM: New Picture Processing API

KMS (Links)

• DRM KMS overview on ST wiki
• Kernel Mode Setting (KMS) in Linux kernel documentation.
• Display Driver Stack from AMD.
• DRM/KMS Kernel Subsystem also from AMD.

Atomic KMS

• Kernel Locking Engineering - Daniel Vetter, Intel
• "Everything Awesome about GPU Drivers" - Daniel Vetter (LCA 2020) (04:41)
• FOSDEM 2015 | Atomic Mode-Setting
• A Journey through Upstream Atomic KMS to Achieve DP Compliance - Manasi Navare, Intel
• XDC2015 - Daniel Vetter - Atomic Modesetting for Drivers
• XDC 2019 | Enabling 8K displays - A story of 33M pixels, 2 CRTCs and no Tears! - Manasi Navare

Atomic KMS (Links)

• Atomic mode setting design overview, part 1
• Atomic mode setting design overview, part 2
• Atomic KMS/DRM on multiple displays is hindered by driver-global serialization of atomic commits

KMS Bridges

• Diving deep into Linux DRM bridge chaining
• Linux DRM Bridge: A Potential Replacer for the DRM Encoder to Standardise a Single API? | Jagan Teki
• Standardising Linux DRM driver implementations by interfacing DRM Bridge as a single API
• XDC 2022 | Runtime display switch implementation in a Linux DRM bridge subsystem | Jagan Teki
• LVC21F-223 Demystifying Linux MIPI DSI Bridge Subsystem

Sample Drivers

VKMS

• XDC 2023 | October 18 | Status Update of the VKMS DRM driver | Maíra Canal and Melissa Wen
• XDC 2021 | Emulating Virtual Hardware in VKMS | Sumera Priyadarsini
• Rodrigo Siqueira & Haneen Mohamed - VKMS
• XDC 2020 | VKMS improvements using IGT GPU Tools

simpledrm

• Trading fbdev for DRM, No Returns Accepted - Geert Uytterhoeven, Glider bv
• openSUSE Conference 2022 - simpledrm - a kernel fbdev replacement
• Modern Graphics from Boot to Shutdown and Retiring fbdev

Rendering Pipeline

Rendering Pipeline

• The Linux 3D graphics stack with Alyssa Rosenzweig
• Open Source Graphics 101: Getting Started - Boris Brezillon, Collabora
• Thomas Zimmermann The Linux Graphics Stack in a Nutshell

DRM (Rendering)

• I want to make a GPU (Playlist)
• Overview of a Modern DRM Driver - Matt Coster, Imagination Technologies
• XDC 2023 | October 19 | With great power comes less responsibility | B. Brezillon, D. Krummrich
• Everything Great about Upstream Graphics - Daniel Vetter, Intel
• Why Are GPUs (Not) Fast - A Trip Through the Driver Stack - Lucas Stach, Pengutronix
• Mesa3D Unveiled: From glDrawArrays to GPU Magic - Christian Gmeiner, Igalia

User Space Components

libdrm

• Getting pixels on screen on Linux: introduction to Kernel Mode Setting - Simon Ser

Wayland Protocol

• Why YOU should write a Wayland compositor – Victoria Brekenfeld – HiP22 Berlin

Mutter

• A Look Inside Mutter / Gnome Shell

Mesa

• Mesa3D Unveiled: From glDrawArrays to GPU Magic - Christian Gmeiner, Igalia
• Embedded Graphics Drivers in Mesa - Neil Roberts, Igalia
• Linux Graphics 101 - Rohan Garg

libinput

• "Write a single library to handle all input devices, it'll be easy" they said…

Tooling

DRM-CI

• Quickly Test Your Kernel with GitLab CI - Helen Koike, Collabora
• Automated testing of the DRM subsystem
• DRM-CI: A GitLab-CI pipeline for Linux kernel testing

IGT GPU Tools

• drm/igt-gpu-tools
• XDC 2020 | VKMS improvements using IGT GPU Tools
• IGT GPU Tools - the past, the present, the future

KUnit

• XDC 2023 | October 17 | Status of KUnit tests on DRM subsystem | Arthur Grillo, Carlos Gallo Filho
• XDC 2022 | KUnit sorcery and the uncanny nature of FPU in the DRM | I. Basso & M. Canal & M. Lemes

apitrace

• apitrace/apitrace
• v3dv: experience using gfxreconstruct/apitrace traces for performance evaluation
• Graphics Performance Analysis with FrameRetrace: A Responsive UI for ApiTrace - Mark Janes, Intel
• Measuring and Improving OpenGL Performance