# Block I/O troubleshoot

- `struct task_struct`
- `struct blk_plug`
- `cb_list`
- `mq_list`
- per-process lock-free queue has constituent per-device request queues
- Generic Block Layer (GBL)
- BIO, `struct bio`
- I/O scheduler
- request queue
- per core sw req queue
- sort, merge request
- tagging
- fairness scheduling
- io accounting
- per block dev dispatch queues
- talks to block device driver

## `blktrace`, `iostat`

- `blktrace`
- works on BIO layer, reports bandwidth, average merge counts and average IO size
- latency at each stage

## `bpftrace`

Use with `profile.py` to generate reports, also framegraphs

Use `stackcount.py` to capture counts of a stack trace, to identify the hot code paths

`argdist` to capture kernel function call args and ret-vals.

## `trace_printk`

Lower overhead. Only dumps to `ftrace` instead of console.

## Issues