[TOC]
## References
Since `kdump` and `crash(8)` are almost always used together (one for *generate* dump, and the other for *analyze* that dump), so references regarding these two utilities are put together.
Other than videos below, [*Kdump in an hour*](https://www.youtube.com/watch?v=XeSqvPj4b48&list=PLMuBvrEUO0NVzIOxXXYiYNWxNQaY-V5J5) is an excellent playlist introducing most commonly used commands `crash(8)`. In fact, this is in my opinion the most friendly tutorial regarding kdump!
### [Postmortem Analysis with Kexec/kdump and Ftrace - Steven Rostedt, Google](https://youtu.be/aUGNDJPpUUg)
{%youtube aUGNDJPpUUg %}
### [Marian Marinov - Analyzing Linux kernel crash dumps](https://youtu.be/wcId2Y9bM-M)
{%youtube wcId2Y9bM-M %}
## Common Usage
Note that there's a `help` command that lists every available command in `crash`.
### `sys`
This the message that prompted when `crash` first load the dump file:
```
KERNEL: /lib/debug/boot/vmlinux-5.19.0-45-generic
DUMPFILE: /var/crash/202306261343/dump.202306261343 [PARTIAL DUMP]
CPUS: 4
DATE: Mon Jun 26 13:43:22 UTC 2023
UPTIME: 00:03:36
LOAD AVERAGE: 1.05, 0.62, 0.27
TASKS: 238
NODENAME: lima-default
RELEASE: 5.19.0-45-generic
VERSION: #46-Ubuntu SMP PREEMPT_DYNAMIC Wed Jun 7 09:08:58 UTC 2023
MACHINE: x86_64 (999 Mhz)
MEMORY: 4 GB
PANIC: "kernel BUG at kernel/entry/common.c:449!"
PID: 0
COMMAND: "swapper/2"
TASK: ffff8be100353080 (1 of 4) [THREAD_INFO: ffff8be100353080]
CPU: 2
STATE: TASK_RUNNING (PANIC)
```
Note that this dump file is captured during me trying to using `trace-cmd` on `__common_interupt` to reproduce results in *Finding Sources of Latency on your Linux System - Steven Rostedt, VMware* on a `lima` virtual machine.
It is likely related to [this recursive breakpoint issue](https://lkml.org/lkml/2023/6/21/1066), which makes tracing trying to tracing trying to tracing trying to tracing trying to tracing trying to....
### `ps`
This pretty much what you would expect from `ps` - listing processes and their information. Note that on SMP machines, the active task on each CPU will be highlighted by an angle bracket (`>`) preceding its information:
```
crash> ps
PID PPID CPU TASK ST %MEM VSZ RSS COMM
> 0 0 0 ffffffff93c1b600 RU 0.0 0 0 [swapper/0]
0 0 1 ffff8be100356100 RU 0.0 0 0 [swapper/1]
> 0 0 2 ffff8be100353080 RU 0.0 0 0 [swapper/2]
> 0 0 3 ffff8be100368000 RU 0.0 0 0 [swapper/3]
1 0 1 ffff8be1002cb080 IN 0.2 169040 13760 systemd
2 0 1 ffff8be1002c9840 IN 0.0 0 0 [kthreadd]
3 2 0 ffff8be1002c8000 ID 0.0 0 0 [rcu_gp]
4 2 0 ffff8be1002cc8c0 ID 0.0 0 0 [rcu_par_gp]
5 2 0 ffff8be1002ce100 ID 0.0 0 0 [slub_flushwq]
...
2477 2422 0 ffff8be104360000 IN 0.1 10908 4988 sudo
2478 2477 3 ffff8be10778e100 IN 0.0 10908 2068 sudo
> 2479 2478 1 ffff8be107dce100 RU 0.1 4880 3424 trace-cmd
```
### `runq`
As its name suggests, show tasks on run queue on each CPU:
```
crash> runq
CPU 0 RUNQUEUE: ffff8be17bc31280
CURRENT: PID: 0 TASK: ffffffff93c1b600 COMMAND: "swapper/0"
RT PRIO_ARRAY: ffff8be17bc31540
[no tasks queued]
CFS RB_ROOT: ffff8be17bc313c0
[no tasks queued]
CPU 1 RUNQUEUE: ffff8be17bcb1280
CURRENT: PID: 2479 TASK: ffff8be107dce100 COMMAND: "trace-cmd"
RT PRIO_ARRAY: ffff8be17bcb1540
[no tasks queued]
CFS RB_ROOT: ffff8be17bcb13c0
[no tasks queued]
CPU 2 RUNQUEUE: ffff8be17bd31280
CURRENT: PID: 0 TASK: ffff8be100353080 COMMAND: "swapper/2"
RT PRIO_ARRAY: ffff8be17bd31540
[no tasks queued]
CFS RB_ROOT: ffff8be17bd313c0
[no tasks queued]
CPU 3 RUNQUEUE: ffff8be17bdb1280
CURRENT: PID: 0 TASK: ffff8be100368000 COMMAND: "swapper/3"
RT PRIO_ARRAY: ffff8be17bdb1540
[no tasks queued]
CFS RB_ROOT: ffff8be17bdb13c0
[no tasks queued]
```
### `bt`
Show backtrace.
```
...
#178 [ffff9f108012fd20] asm_exc_int3 at ffffffff92c00b4a
#179 [ffff9f108012fd90] exc_int3 at ffffffff92b2b475
#180 [ffff9f108012fe88] do_int3 at ffffffff91c428f5
#181 [ffff9f108012feb8] do_int3 at ffffffff91c428f5
#182 [ffff9f108012fec8] exc_int3 at ffffffff92b2b475
#183 [ffff9f108012fed8] do_int3 at ffffffff91c428f5
#184 [ffff9f108012fee0] exc_int3 at ffffffff92b2b475
#185 [ffff9f108012ff00] asm_exc_int3 at ffffffff92c00b4a
#186 [ffff9f108012ff70] exc_int3 at ffffffff92b2b475
#187 [ffff9f1080130068] do_int3 at ffffffff91c428f5
#188 [ffff9f1080130098] do_int3 at ffffffff91c428f5
#189 [ffff9f10801300a8] exc_int3 at ffffffff92b2b475
#190 [ffff9f10801300b8] do_int3 at ffffffff91c428f5
#191 [ffff9f10801300c0] exc_int3 at ffffffff92b2b475
#192 [ffff9f10801300e0] asm_exc_int3 at ffffffff92c00b4a
#193 [ffff9f1080130150] exc_int3 at ffffffff92b2b475
#194 [ffff9f1080130248] do_int3 at ffffffff91c428f5
#195 [ffff9f1080130278] do_int3 at ffffffff91c428f5
#196 [ffff9f1080130288] exc_int3 at ffffffff92b2b475
#197 [ffff9f1080130298] do_int3 at ffffffff91c428f5
#198 [ffff9f10801302a0] exc_int3 at ffffffff92b2b475
#199 [ffff9f10801302c0] asm_exc_int3 at ffffffff92c00b4a
#200 [ffff9f1080130330] exc_int3 at ffffffff92b2b475
#201 [ffff9f1080130428] do_int3 at ffffffff91c428f5
#202 [ffff9f1080130458] do_int3 at ffffffff91c428f5
#203 [ffff9f1080130468] exc_int3 at ffffffff92b2b475
#204 [ffff9f1080130478] do_int3 at ffffffff91c428f5
#205 [ffff9f1080130480] exc_int3 at ffffffff92b2b475
#206 [ffff9f10801304a0] asm_exc_int3 at ffffffff92c00b4a
...
```
### `kmem`
Show memory statistics. For example:
```
crash> kmem -i
PAGES TOTAL PERCENTAGE
TOTAL MEM 901896 3.4 GB ----
FREE 608797 2.3 GB 67% of TOTAL MEM
USED 293099 1.1 GB 32% of TOTAL MEM
SHARED 64608 252.4 MB 7% of TOTAL MEM
BUFFERS 6519 25.5 MB 0% of TOTAL MEM
CACHED 215277 840.9 MB 23% of TOTAL MEM
SLAB 24001 93.8 MB 2% of TOTAL MEM
TOTAL HUGE 0 0 ----
HUGE FREE 0 0 0% of TOTAL HUGE
TOTAL SWAP 0 0 ----
SWAP USED 0 0 0% of TOTAL SWAP
SWAP FREE 0 0 0% of TOTAL SWAP
COMMIT LIMIT 450948 1.7 GB ----
COMMITTED 215601 842.2 MB 47% of TOTAL LIMIT
```
Sign in with Wallet
Connect another wallet