# Kernel Debugging References - trace-cmd

[TOC]

## References

Basically try search *Steven Rostedt* on YouTube and watch his every talk.

### [Steven Rostedt - Learning the Linux Kernel with tracing](https://youtu.be/JRyrhsx-L5Y)

{%youtube JRyrhsx-L5Y %}

### [Tracing Summit 2022 - State of the Linux Tracers (34:43)](https://youtu.be/Er8q8Vtp9eM?t=2083)

{%youtube Er8q8Vtp9eM %}

### [Tracing VMs Seeing How Host and Guest Interact - Steven Rostedt, Google](https://youtu.be/v0ocveEsvNU)

{%youtube v0ocveEsvNU %}

### [Kernel Recipes 2022 - Ftrace update](https://youtu.be/CLUVleQBm-w)

{%youtube CLUVleQBm-w %}

### [Embedded Recipes 2022 - Tracing on embedded boards](https://youtu.be/7KHLoZKwlBk)

{%youtube 7KHLoZKwlBk %}

### [Finding Sources of Latency on your Linux System - Steven Rostedt, VMware (18:50 ~ FIN)](https://youtu.be/Tkra8g0gXAU?t=1130)

{%youtube Tkra8g0gXAU %}

Note that `do_IRQ` has been replaced with [`common_interrupt`](https://elixir.bootlin.com/linux/latest/source/arch/x86/kernel/irq.c#L240) in x86_64 architecture. See this mailing list regarding [the change](https://lore.kernel.org/lkml/[e-mail]/t/) made in documentation. Hook into `irq_handler_entry` could see its call stack. This equivalent would be:

$ trace-cmd record \
    -p function_graph \
    -g __common_interrupt \
    -g '*_interrupt' \
    -e irq_handler_entry

For 5.19 kernel, it go through the following code path:

    __handle_irq_event_percpu+305
    __handle_irq_event_percpu+305
    handle_irq_event+57
    handle_edge_irq+140
    __common_interrupt+86
    common_interrupt+68
    asm_common_interrupt+39

## Cheat Sheet

### List tracepoints

List tracepoints:

$ trace-cmd list -e

List tracepoint field:

trace-cmd list -F -e "syscalls:sys_enter_nanosleep"

List tracepoint format with, with format string:

trace-cmd list -F --full -e "syscalls:sys_enter_nanosleep"

Most tracepoints are from `/sys/kernel/tracing/available_events`

### List tracable functions

List available filter function:

$ trace-cmd list -f

### Trace function call stack

$ trace-cmd record -p function --func-stack -l "__common_interrupt"

### Trace function execution

$ trace-cmd record -p function_graph -g "__common_interrupt"