# Linux AIO Randrw Benchmark

Benchmark: FIO with 50Gb file, randrw workload, 4K blocksize, direct, iodepth 32
System:
- Corsair MP600 PRO LPX 4TB
- Linux 6.1.0-17amd64
- AMD Ryzen 9 7950X 16-Core Processor

| Filesystem | IO Interface | Read | Write | Latency | IOPS |
| -------- | -------- | -------- | ----- | ----- | ---- |
| Ext4 | posixaio | 53.8 MiB/s | 54.2 MiB/s | 1100us | 13.8k |
| Ext4 | io_submit | 576 MiB/s | 572 MiB/s | 135us | 146k |
| Ext4 | io_uring | 674 MiB/s | 670 MiB/s | 135us | 171k |
| None | posixaio | 246 MiB/s | 246 MiB/s | 267us | 63k |
| None | io_submit | 1370 MiB/s | 1369 MiB/s | 49us | 350k |
| None | io_uring | 1636 MiB/s | 1630 MiB/s | 40us | 417k |
| XFS | posixaio | 58 MiB/s | 57.4 MiB/s | 1100us | 14.7k |
| XFS | io_submit | 790 MiB/s | 794 MiB/s | 117us | 202k |
| XFS | io_uring | 721 MiB/s | 719 MiB/s | 114us | 184k |

---

Ext4 posixaio randrw

fio --name=fiotest --filename=test --size=50Gb --rw=randrw --bs=4Kb --direct=1 --ioengine=posixaio
--iodepth=32 --startdelay=5 --group_reporting --runtime=30

Jobs: 1 (f=1): [m(1)][100.0%][r=53.8MiB/s,w=54.2MiB/s][r=13.8k,w=13.9k IOPS][eta 00m:00s]
fiotest: (groupid=0, jobs=1): err= 0: pid=2934581: Fri Mar 29 10:25:30 2024
  read: IOPS=14.3k, BW=56.0MiB/s (58.7MB/s)(1680MiB/30001msec)
    slat (nsec): min=30, max=61494, avg=155.41, stdev=165.21
    clat (usec): min=270, max=69252, avg=1141.25, stdev=1374.54
     lat (usec): min=271, max=69253, avg=1141.41, stdev=1374.55
    clat percentiles (usec):
     |  1.00th=[  725],  5.00th=[  824], 10.00th=[  881], 20.00th=[  938],
     | 30.00th=[  988], 40.00th=[ 1029], 50.00th=[ 1074], 60.00th=[ 1106],
     | 70.00th=[ 1139], 80.00th=[ 1188], 90.00th=[ 1254], 95.00th=[ 1336],
     | 99.00th=[ 1500], 99.50th=[ 4146], 99.90th=[24511], 99.95th=[38011],
     | 99.99th=[53216]
   bw (  KiB/s): min=33888, max=61112, per=100.00%, avg=57416.14, stdev=5068.32, samples=59
   iops        : min= 8472, max=15278, avg=14354.03, stdev=1267.08, samples=59
  write: IOPS=14.3k, BW=55.9MiB/s (58.7MB/s)(1678MiB/30001msec); 0 zone resets
    slat (nsec): min=40, max=6602, avg=250.01, stdev=146.98
    clat (usec): min=150, max=69251, avg=1085.40, stdev=1378.77
     lat (usec): min=150, max=69251, avg=1085.65, stdev=1378.77
    clat percentiles (usec):
     |  1.00th=[  668],  5.00th=[  775], 10.00th=[  824], 20.00th=[  881],
     | 30.00th=[  930], 40.00th=[  979], 50.00th=[ 1029], 60.00th=[ 1045],
     | 70.00th=[ 1090], 80.00th=[ 1139], 90.00th=[ 1205], 95.00th=[ 1254],
     | 99.00th=[ 1401], 99.50th=[ 3884], 99.90th=[23725], 99.95th=[38011],
     | 99.99th=[53740]
   bw (  KiB/s): min=33688, max=61768, per=100.00%, avg=57340.75, stdev=5142.19, samples=59
   iops        : min= 8422, max=15442, avg=14335.19, stdev=1285.55, samples=59
  lat (usec)   : 250=0.01%, 500=0.04%, 750=2.27%, 1000=37.79%
  lat (msec)   : 2=59.25%, 4=0.14%, 10=0.29%, 20=0.08%, 50=0.11%
  lat (msec)   : 100=0.01%
  cpu          : usr=2.80%, sys=0.79%, ctx=198064, majf=0, minf=26
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=3.6%, 16=70.6%, 32=25.8%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=94.6%, 8=1.7%, 16=2.9%, 32=0.8%, 64=0.0%, >=64=0.0%
     issued rwts: total=430042,429602,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=56.0MiB/s (58.7MB/s), 56.0MiB/s-56.0MiB/s (58.7MB/s-58.7MB/s), io=1680MiB (1761MB), run=30001-30001msec
  WRITE: bw=55.9MiB/s (58.7MB/s), 55.9MiB/s-55.9MiB/s (58.7MB/s-58.7MB/s), io=1678MiB (1760MB), run=30001-30001msec

Disk stats (read/write):
  nvme0n1: ios=429092/429427, merge=0/1462, ticks=25152/4540, in_queue=30485, util=99.72%

----

Ext4 io_uring randrw

fio --name=fiotest --filename=test --size=50Gb --rw=randrw --bs=4Kb --direct=1 --ioengine=io_uring
--iodepth=32 --startdelay=5 --group_reporting --runtime=30

Jobs: 1 (f=1): [m(1)][100.0%][r=674MiB/s,w=670MiB/s][r=172k,w=171k IOPS][eta 00m:00s]
fiotest: (groupid=0, jobs=1): err= 0: pid=2935018: Fri Mar 29 10:27:19 2024
  read: IOPS=170k, BW=663MiB/s (695MB/s)(19.4GiB/30001msec)
    slat (nsec): min=811, max=129109, avg=2266.70, stdev=1217.41
    clat (usec): min=34, max=20095, avg=130.96, stdev=196.83
     lat (usec): min=57, max=20097, avg=133.23, stdev=196.86
    clat percentiles (usec):
     |  1.00th=[   78],  5.00th=[   86], 10.00th=[   90], 20.00th=[   97],
     | 30.00th=[  102], 40.00th=[  109], 50.00th=[  115], 60.00th=[  124],
     | 70.00th=[  135], 80.00th=[  147], 90.00th=[  169], 95.00th=[  192],
     | 99.00th=[  249], 99.50th=[  293], 99.90th=[ 2638], 99.95th=[ 5014],
     | 99.99th=[ 7767]
   bw (  KiB/s): min=234672, max=805312, per=99.87%, avg=677954.81, stdev=102958.15, samples=59
   iops        : min=58668, max=201328, avg=169488.69, stdev=25739.57, samples=59
  write: IOPS=170k, BW=663MiB/s (695MB/s)(19.4GiB/30001msec); 0 zone resets
    slat (nsec): min=901, max=165227, avg=2498.21, stdev=1283.23
    clat (usec): min=4, max=20468, avg=52.39, stdev=243.30
     lat (usec): min=7, max=20470, avg=54.89, stdev=243.35
    clat percentiles (usec):
     |  1.00th=[   18],  5.00th=[   23], 10.00th=[   26], 20.00th=[   31],
     | 30.00th=[   35], 40.00th=[   38], 50.00th=[   41], 60.00th=[   45],
     | 70.00th=[   50], 80.00th=[   57], 90.00th=[   74], 95.00th=[   85],
     | 99.00th=[  104], 99.50th=[  113], 99.90th=[ 2474], 99.95th=[ 6194],
     | 99.99th=[10814]
   bw (  KiB/s): min=233800, max=805976, per=99.86%, avg=677566.42, stdev=102850.81, samples=59
   iops        : min=58450, max=201494, avg=169391.59, stdev=25712.71, samples=59
  lat (usec)   : 10=0.02%, 20=1.29%, 50=34.39%, 100=26.81%, 250=36.94%
  lat (usec)   : 500=0.35%, 750=0.02%, 1000=0.01%
  lat (msec)   : 2=0.02%, 4=0.06%, 10=0.07%, 20=0.01%, 50=0.01%
  cpu          : usr=12.41%, sys=70.79%, ctx=1671167, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=5091653,5089211,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=663MiB/s (695MB/s), 663MiB/s-663MiB/s (695MB/s-695MB/s), io=19.4GiB (20.9GB), run=30001-30001msec
  WRITE: bw=663MiB/s (695MB/s), 663MiB/s-663MiB/s (695MB/s-695MB/s), io=19.4GiB (20.8GB), run=30001-30001msec

Disk stats (read/write):
  nvme0n1: ios=5051855/5049663, merge=0/1486, ticks=483495/109994, in_queue=594396, util=99.73%

----

Ext4 io_submit randrw

fio --name=fiotest --filename=test --size=50Gb --rw=randrw --bs=4Kb --direct=1 --ioengine=libaio
--iodepth=32 --startdelay=5 --group_reporting --runtime=30

obs: 1 (f=1): [m(1)][100.0%][r=576MiB/s,w=572MiB/s][r=148k,w=146k IOPS][eta 00m:00s] 
fiotest: (groupid=0, jobs=1): err= 0: pid=2936145: Fri Mar 29 10:32:22 2024
  read: IOPS=166k, BW=650MiB/s (682MB/s)(19.1GiB/30001msec)
    slat (nsec): min=822, max=202576, avg=2165.57, stdev=1198.00
    clat (usec): min=37, max=19221, avg=125.27, stdev=176.12
     lat (usec): min=57, max=19226, avg=127.44, stdev=176.16
    clat percentiles (usec):
     |  1.00th=[   70],  5.00th=[   78], 10.00th=[   84], 20.00th=[   92],
     | 30.00th=[   98], 40.00th=[  104], 50.00th=[  112], 60.00th=[  120],
     | 70.00th=[  130], 80.00th=[  143], 90.00th=[  165], 95.00th=[  186],
     | 99.00th=[  241], 99.50th=[  285], 99.90th=[ 2540], 99.95th=[ 4948],
     | 99.99th=[ 6718]
   bw (  KiB/s): min=449312, max=775520, per=100.00%, avg=666403.39, stdev=75541.67, samples=59
   iops        : min=112328, max=193880, avg=166600.92, stdev=18885.52, samples=59
  write: IOPS=166k, BW=650MiB/s (681MB/s)(19.0GiB/30001msec); 0 zone resets
    slat (nsec): min=891, max=195353, avg=2386.02, stdev=1238.89
    clat (usec): min=4, max=19825, avg=61.95, stdev=230.38
     lat (usec): min=7, max=19829, avg=64.34, stdev=230.44
    clat percentiles (usec):
     |  1.00th=[   22],  5.00th=[   31], 10.00th=[   35], 20.00th=[   41],
     | 30.00th=[   45], 40.00th=[   49], 50.00th=[   53], 60.00th=[   58],
     | 70.00th=[   63], 80.00th=[   70], 90.00th=[   80], 95.00th=[   91],
     | 99.00th=[  122], 99.50th=[  133], 99.90th=[ 1156], 99.95th=[ 5604],
     | 99.99th=[10945]
   bw (  KiB/s): min=447904, max=775768, per=100.00%, avg=665998.37, stdev=75795.00, samples=59
   iops        : min=111976, max=193942, avg=166499.59, stdev=18948.77, samples=59
  lat (usec)   : 10=0.02%, 20=0.27%, 50=21.27%, 100=43.63%, 250=34.33%
  lat (usec)   : 500=0.28%, 750=0.02%, 1000=0.01%
  lat (msec)   : 2=0.03%, 4=0.06%, 10=0.05%, 20=0.01%
  cpu          : usr=14.16%, sys=70.17%, ctx=1758061, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=4994059,4991118,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=650MiB/s (682MB/s), 650MiB/s-650MiB/s (682MB/s-682MB/s), io=19.1GiB (20.5GB), run=30001-30001msec
  WRITE: bw=650MiB/s (681MB/s), 650MiB/s-650MiB/s (681MB/s-681MB/s), io=19.0GiB (20.4GB), run=30001-30001msec

Disk stats (read/write):
  nvme0n1: ios=4986358/4984167, merge=0/1348, ticks=465026/102022, in_queue=567756, util=99.98%

----

Block device, posixaio randrw

fio --name=fiotest --filename=/dev/nvme0n1p5 --size=50Gb --rw=randrw --bs=4Kb --direct=1 --ioengine=posixaio
--iodepth=32 --startdelay=5 --group_reporting --runtime=30

Jobs: 1 (f=1): [m(1)][100.0%][r=246MiB/s,w=246MiB/s][r=63.0k,w=62.9k IOPS][eta 00m:00s]
fiotest: (groupid=0, jobs=1): err= 0: pid=2933522: Fri Mar 29 10:22:10 2024
  read: IOPS=59.1k, BW=231MiB/s (242MB/s)(6924MiB/30001msec)
    slat (nsec): min=30, max=58027, avg=166.66, stdev=184.95
    clat (usec): min=10, max=14040, avg=266.91, stdev=100.13
     lat (usec): min=10, max=14040, avg=267.08, stdev=100.14
    clat percentiles (usec):
     |  1.00th=[  235],  5.00th=[  241], 10.00th=[  243], 20.00th=[  245],
     | 30.00th=[  249], 40.00th=[  251], 50.00th=[  253], 60.00th=[  255],
     | 70.00th=[  258], 80.00th=[  262], 90.00th=[  269], 95.00th=[  293],
     | 99.00th=[  898], 99.50th=[  914], 99.90th=[  955], 99.95th=[ 1057],
     | 99.99th=[ 2311]
   bw (  KiB/s): min=70104, max=255232, per=99.93%, avg=236161.08, stdev=35473.31, samples=59
   iops        : min=17526, max=63808, avg=59040.27, stdev=8868.33, samples=59
  write: IOPS=59.0k, BW=231MiB/s (242MB/s)(6917MiB/30001msec); 0 zone resets
    slat (nsec): min=30, max=12874, avg=263.62, stdev=186.10
    clat (usec): min=9, max=14038, avg=265.65, stdev=95.64
     lat (usec): min=10, max=14039, avg=265.91, stdev=95.65
    clat percentiles (usec):
     |  1.00th=[  233],  5.00th=[  239], 10.00th=[  241], 20.00th=[  243],
     | 30.00th=[  247], 40.00th=[  249], 50.00th=[  251], 60.00th=[  253],
     | 70.00th=[  258], 80.00th=[  262], 90.00th=[  269], 95.00th=[  293],
     | 99.00th=[  898], 99.50th=[  914], 99.90th=[  955], 99.95th=[ 1057],
     | 99.99th=[ 1909]
   bw (  KiB/s): min=68664, max=255232, per=99.93%, avg=235917.83, stdev=35629.19, samples=59
   iops        : min=17166, max=63808, avg=58979.46, stdev=8907.30, samples=59
  lat (usec)   : 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%, 250=43.67%
  lat (usec)   : 500=54.00%, 750=1.07%, 1000=1.19%
  lat (msec)   : 2=0.06%, 4=0.01%, 10=0.01%, 20=0.01%
  cpu          : usr=10.62%, sys=2.92%, ctx=742109, majf=0, minf=26
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=16.3%, 16=59.0%, 32=24.7%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=96.6%, 8=0.5%, 16=0.8%, 32=2.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=1772502,1770691,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=231MiB/s (242MB/s), 231MiB/s-231MiB/s (242MB/s-242MB/s), io=6924MiB (7260MB), run=30001-30001msec
  WRITE: bw=231MiB/s (242MB/s), 231MiB/s-231MiB/s (242MB/s-242MB/s), io=6917MiB (7253MB), run=30001-30001msec

Disk stats (read/write):
  nvme0n1: ios=1766205/1765029, merge=0/1122, ticks=13303/11069, in_queue=24395, util=99.75%

-----

block device, io_uring randrw

fio --name=fiotest --filename=/dev/nvme0n1p5 --size=50Gb --rw=randrw --bs=4Kb --direct=1 --ioengine=io_uring
--iodepth=32 --startdelay=5 --group_reporting --runtime=30

fiotest: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=32
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [m(1)][90.9%][r=1636MiB/s,w=1627MiB/s][r=419k,w=417k IOPS][eta 00m:02s]
fiotest: (groupid=0, jobs=1): err= 0: pid=2934097: Fri Mar 29 10:23:33 2024
  read: IOPS=406k, BW=1588MiB/s (1665MB/s)(23.3GiB/15021msec)
    slat (nsec): min=611, max=16571, avg=858.12, stdev=468.41
    clat (usec): min=7, max=8239, avg=40.73, stdev=52.82
     lat (usec): min=8, max=8252, avg=41.59, stdev=52.82
    clat percentiles (usec):
     |  1.00th=[   36],  5.00th=[   37], 10.00th=[   37], 20.00th=[   38],
     | 30.00th=[   38], 40.00th=[   39], 50.00th=[   39], 60.00th=[   40],
     | 70.00th=[   41], 80.00th=[   41], 90.00th=[   44], 95.00th=[   49],
     | 99.00th=[   56], 99.50th=[   65], 99.90th=[  169], 99.95th=[  182],
     | 99.99th=[ 4359]
   bw (  MiB/s): min= 1114, max= 1657, per=100.00%, avg=1588.38, stdev=127.90, samples=30
   iops        : min=285194, max=424242, avg=406626.00, stdev=32742.13, samples=30
  write: IOPS=406k, BW=1587MiB/s (1664MB/s)(23.3GiB/15021msec); 0 zone resets
    slat (nsec): min=651, max=19086, avg=1037.85, stdev=525.84
    clat (usec): min=4, max=8252, avg=35.75, stdev=54.70
     lat (usec): min=6, max=8253, avg=36.79, stdev=54.71
    clat percentiles (usec):
     |  1.00th=[   28],  5.00th=[   31], 10.00th=[   32], 20.00th=[   33],
     | 30.00th=[   34], 40.00th=[   35], 50.00th=[   35], 60.00th=[   36],
     | 70.00th=[   36], 80.00th=[   37], 90.00th=[   39], 95.00th=[   45],
     | 99.00th=[   50], 99.50th=[   51], 99.90th=[   62], 99.95th=[  192],
     | 99.99th=[ 4359]
   bw (  MiB/s): min= 1115, max= 1653, per=100.00%, avg=1587.43, stdev=127.90, samples=30
   iops        : min=285562, max=423398, avg=406381.33, stdev=32743.10, samples=30
  lat (usec)   : 10=0.04%, 20=0.07%, 50=97.65%, 100=1.98%, 250=0.23%
  lat (usec)   : 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%
  cpu          : usr=29.97%, sys=67.90%, ctx=3646, majf=0, minf=15
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=6105354,6101750,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=1588MiB/s (1665MB/s), 1588MiB/s-1588MiB/s (1665MB/s-1665MB/s), io=23.3GiB (25.0GB), run=15021-15021msec
  WRITE: bw=1587MiB/s (1664MB/s), 1587MiB/s-1587MiB/s (1664MB/s-1664MB/s), io=23.3GiB (25.0GB), run=15021-15021msec

Disk stats (read/write):
  nvme0n1: ios=6053653/6050055, merge=0/714, ticks=98487/69034, in_queue=167539, util=99.41%

-----

block device io_submit randrw

fio --name=fiotest --filename=/dev/nvme0n1p5 --size=50Gb --rw=randrw --bs=4Kb --direct=1 --ioengine=libaio 
--iodepth=32 --startdelay=5 --group_reporting --runtime=30

Jobs: 1 (f=1): [m(1)][92.3%][r=1370MiB/s,w=1369MiB/s][r=351k,w=350k IOPS][eta 00m:02s]
fiotest: (groupid=0, jobs=1): err= 0: pid=2936738: Fri Mar 29 10:33:25 2024
  read: IOPS=325k, BW=1270MiB/s (1332MB/s)(23.3GiB/18772msec)
    slat (nsec): min=641, max=24495, avg=879.21, stdev=418.10
    clat (usec): min=8, max=20170, avg=48.64, stdev=152.32
     lat (usec): min=9, max=20171, avg=49.52, stdev=152.33
    clat percentiles (usec):
     |  1.00th=[   31],  5.00th=[   33], 10.00th=[   35], 20.00th=[   38],
     | 30.00th=[   42], 40.00th=[   45], 50.00th=[   47], 60.00th=[   49],
     | 70.00th=[   51], 80.00th=[   54], 90.00th=[   56], 95.00th=[   59],
     | 99.00th=[   71], 99.50th=[   82], 99.90th=[  115], 99.95th=[  635],
     | 99.99th=[ 6849]
   bw (  MiB/s): min=  481, max= 1436, per=99.89%, avg=1269.09, stdev=232.51, samples=37
   iops        : min=123266, max=367704, avg=324887.73, stdev=59522.83, samples=37
  write: IOPS=325k, BW=1270MiB/s (1331MB/s)(23.3GiB/18772msec); 0 zone resets
    slat (nsec): min=691, max=78135, avg=1048.32, stdev=469.87
    clat (usec): min=6, max=20170, avg=47.41, stdev=214.44
     lat (usec): min=7, max=20171, avg=48.46, stdev=214.45
    clat percentiles (usec):
     |  1.00th=[   27],  5.00th=[   29], 10.00th=[   31], 20.00th=[   34],
     | 30.00th=[   37], 40.00th=[   40], 50.00th=[   43], 60.00th=[   46],
     | 70.00th=[   49], 80.00th=[   51], 90.00th=[   54], 95.00th=[   56],
     | 99.00th=[   69], 99.50th=[   78], 99.90th=[  429], 99.95th=[ 3916],
     | 99.99th=[12518]
   bw (  MiB/s): min=  480, max= 1431, per=99.88%, avg=1268.21, stdev=231.43, samples=37
   iops        : min=122960, max=366520, avg=324661.68, stdev=59246.93, samples=37
  lat (usec)   : 10=0.01%, 20=0.01%, 50=70.77%, 100=29.08%, 250=0.05%
  lat (usec)   : 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.02%, 10=0.03%, 20=0.01%, 50=0.01%
  cpu          : usr=44.31%, sys=48.06%, ctx=1341, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=6105354,6101750,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=1270MiB/s (1332MB/s), 1270MiB/s-1270MiB/s (1332MB/s-1332MB/s), io=23.3GiB (25.0GB), run=18772-18772msec
  WRITE: bw=1270MiB/s (1331MB/s), 1270MiB/s-1270MiB/s (1331MB/s-1331MB/s), io=23.3GiB (25.0GB), run=18772-18772msec

Disk stats (read/write):
  nvme0n1: ios=6060079/6056563, merge=0/927, ticks=106744/98562, in_queue=206017, util=99.55%

----

XFS posixaio randrw

fio --name=fiotest --filename=test1 --size=50Gb --rw=randrw --bs=4Kb --direct=1 --ioengine=posixaio --iodepth=32 --startd
elay=5 --group_reporting --runtime=30

Jobs: 1 (f=1): [m(1)][100.0%][r=58.0MiB/s,w=57.4MiB/s][r=14.9k,w=14.7k IOPS][eta 00m:00s]
fiotest: (groupid=0, jobs=1): err= 0: pid=2943690: Fri Mar 29 12:51:34 2024
  read: IOPS=14.1k, BW=55.2MiB/s (57.9MB/s)(1657MiB/30002msec)
    slat (nsec): min=30, max=70081, avg=151.25, stdev=170.43
    clat (usec): min=365, max=65106, avg=1159.02, stdev=1362.58
     lat (usec): min=365, max=65106, avg=1159.17, stdev=1362.58
    clat percentiles (usec):
     |  1.00th=[  750],  5.00th=[  840], 10.00th=[  889], 20.00th=[  971],
     | 30.00th=[ 1020], 40.00th=[ 1045], 50.00th=[ 1090], 60.00th=[ 1139],
     | 70.00th=[ 1172], 80.00th=[ 1221], 90.00th=[ 1287], 95.00th=[ 1352],
     | 99.00th=[ 1516], 99.50th=[ 3621], 99.90th=[25560], 99.95th=[37487],
     | 99.99th=[50594]
   bw (  KiB/s): min=34544, max=60304, per=99.96%, avg=56536.95, stdev=5126.73, samples=59
   iops        : min= 8636, max=15076, avg=14134.24, stdev=1281.68, samples=59
  write: IOPS=14.1k, BW=55.2MiB/s (57.8MB/s)(1655MiB/30002msec); 0 zone resets
    slat (nsec): min=40, max=62197, avg=248.87, stdev=183.16
    clat (usec): min=209, max=65515, avg=1098.81, stdev=1299.25
     lat (usec): min=210, max=65515, avg=1099.06, stdev=1299.25
    clat percentiles (usec):
     |  1.00th=[  676],  5.00th=[  783], 10.00th=[  840], 20.00th=[  914],
     | 30.00th=[  963], 40.00th=[  988], 50.00th=[ 1037], 60.00th=[ 1074],
     | 70.00th=[ 1123], 80.00th=[ 1172], 90.00th=[ 1237], 95.00th=[ 1287],
     | 99.00th=[ 1434], 99.50th=[ 3163], 99.90th=[23725], 99.95th=[36963],
     | 99.99th=[49546]
   bw (  KiB/s): min=34584, max=60976, per=99.97%, avg=56470.24, stdev=5189.04, samples=59
   iops        : min= 8646, max=15244, avg=14117.56, stdev=1297.26, samples=59
  lat (usec)   : 250=0.01%, 500=0.02%, 750=1.88%, 1000=33.03%
  lat (msec)   : 2=64.50%, 4=0.12%, 10=0.27%, 20=0.06%, 50=0.10%
  lat (msec)   : 100=0.01%
  cpu          : usr=3.00%, sys=0.55%, ctx=197083, majf=0, minf=27
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=1.9%, 16=72.2%, 32=25.9%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=94.4%, 8=2.3%, 16=2.8%, 32=0.5%, 64=0.0%, >=64=0.0%
     issued rwts: total=424236,423676,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=55.2MiB/s (57.9MB/s), 55.2MiB/s-55.2MiB/s (57.9MB/s-57.9MB/s), io=1657MiB (1738MB), run=30002-30002msec
  WRITE: bw=55.2MiB/s (57.8MB/s), 55.2MiB/s-55.2MiB/s (57.8MB/s-57.8MB/s), io=1655MiB (1735MB), run=30002-30002msec

Disk stats (read/write):
  nvme0n1: ios=422983/423090, merge=0/1067, ticks=25280/4273, in_queue=30239, util=99.70%

---

XFS libaio randrw

fio --name=fiotest --filename=test2 --size=50Gb --rw=randrw --bs=4Kb --direct=1 --ioengine=libaio --iodepth=32 --startd
elay=5 --group_reporting --runtime=30

Jobs: 1 (f=1): [m(1)][100.0%][r=790MiB/s,w=794MiB/s][r=202k,w=203k IOPS][eta 00m:00s] 
fiotest: (groupid=0, jobs=1): err= 0: pid=2944232: Fri Mar 29 12:53:37 2024
  read: IOPS=205k, BW=800MiB/s (839MB/s)(23.4GiB/30001msec)
    slat (nsec): min=801, max=67636, avg=1247.29, stdev=920.00
    clat (usec): min=54, max=19706, avg=115.94, stdev=173.82
     lat (usec): min=56, max=19709, avg=117.19, stdev=173.84
    clat percentiles (usec):
     |  1.00th=[   63],  5.00th=[   69], 10.00th=[   74], 20.00th=[   81],
     | 30.00th=[   87], 40.00th=[   93], 50.00th=[  100], 60.00th=[  109],
     | 70.00th=[  120], 80.00th=[  137], 90.00th=[  163], 95.00th=[  188],
     | 99.00th=[  249], 99.50th=[  285], 99.90th=[ 2474], 99.95th=[ 4883],
     | 99.99th=[ 6718]
   bw (  KiB/s): min=461376, max=913736, per=100.00%, avg=819612.88, stdev=94874.25, samples=59
   iops        : min=115344, max=228436, avg=204903.29, stdev=23718.52, samples=59
  write: IOPS=205k, BW=799MiB/s (838MB/s)(23.4GiB/30001msec); 0 zone resets
    slat (nsec): min=811, max=73898, avg=1393.44, stdev=980.00
    clat (usec): min=4, max=20869, avg=37.22, stdev=197.44
     lat (usec): min=7, max=20872, avg=38.61, stdev=197.48
    clat percentiles (usec):
     |  1.00th=[   11],  5.00th=[   14], 10.00th=[   17], 20.00th=[   21],
     | 30.00th=[   24], 40.00th=[   28], 50.00th=[   31], 60.00th=[   34],
     | 70.00th=[   37], 80.00th=[   42], 90.00th=[   51], 95.00th=[   64],
     | 99.00th=[   87], 99.50th=[   97], 99.90th=[  130], 99.95th=[ 4883],
     | 99.99th=[10028]
   bw (  KiB/s): min=463680, max=912104, per=100.00%, avg=819102.64, stdev=94827.77, samples=59
   iops        : min=115920, max=228026, avg=204775.66, stdev=23706.94, samples=59
  lat (usec)   : 10=0.42%, 20=9.05%, 50=35.23%, 100=30.45%, 250=24.32%
  lat (usec)   : 500=0.38%, 750=0.02%, 1000=0.01%
  lat (msec)   : 2=0.02%, 4=0.05%, 10=0.05%, 20=0.01%, 50=0.01%
  cpu          : usr=17.94%, sys=52.43%, ctx=2543326, majf=0, minf=15
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=6142688,6138998,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=800MiB/s (839MB/s), 800MiB/s-800MiB/s (839MB/s-839MB/s), io=23.4GiB (25.2GB), run=30001-30001msec
  WRITE: bw=799MiB/s (838MB/s), 799MiB/s-799MiB/s (838MB/s-838MB/s), io=23.4GiB (25.1GB), run=30001-30001msec

Disk stats (read/write):
  nvme0n1: ios=6110245/6107586, merge=5/1059, ticks=617122/110697, in_queue=728504, util=99.70%

---

XFS io_uring randrw

fio --name=fiotest --filename=test3 --size=50Gb --rw=randrw --bs=4Kb --direct=1 --ioengine=io_uring --iodepth=32 --startd
elay=5 --group_reporting --runtime=30
Jobs: 1 (f=1): [m(1)][100.0%][r=721MiB/s,w=719MiB/s][r=185k,w=184k IOPS][eta 00m:00s] 
fiotest: (groupid=0, jobs=1): err= 0: pid=2944731: Fri Mar 29 12:55:11 2024
  read: IOPS=221k, BW=864MiB/s (906MB/s)(25.0GiB/29642msec)
    slat (nsec): min=661, max=661616, avg=1110.41, stdev=926.02
    clat (usec): min=26, max=17707, avg=113.77, stdev=124.02
     lat (usec): min=57, max=17708, avg=114.88, stdev=124.03
    clat percentiles (usec):
     |  1.00th=[   69],  5.00th=[   74], 10.00th=[   77], 20.00th=[   81],
     | 30.00th=[   85], 40.00th=[   91], 50.00th=[   98], 60.00th=[  109],
     | 70.00th=[  120], 80.00th=[  137], 90.00th=[  165], 95.00th=[  190],
     | 99.00th=[  247], 99.50th=[  277], 99.90th=[  750], 99.95th=[ 2278],
     | 99.99th=[ 5145]
   bw (  KiB/s): min=342320, max=937672, per=100.00%, avg=884847.86, stdev=91056.03, samples=59
   iops        : min=85580, max=234420, avg=221212.03, stdev=22764.03, samples=59
  write: IOPS=221k, BW=863MiB/s (905MB/s)(25.0GiB/29642msec); 0 zone resets
    slat (nsec): min=441, max=137657, avg=1259.69, stdev=936.03
    clat (nsec): min=1032, max=15523k, avg=28137.17, stdev=116080.06
     lat (usec): min=7, max=15523, avg=29.40, stdev=116.11
    clat percentiles (usec):
     |  1.00th=[   12],  5.00th=[   15], 10.00th=[   16], 20.00th=[   18],
     | 30.00th=[   20], 40.00th=[   22], 50.00th=[   24], 60.00th=[   26],
     | 70.00th=[   29], 80.00th=[   30], 90.00th=[   36], 95.00th=[   46],
     | 99.00th=[   68], 99.50th=[   79], 99.90th=[  906], 99.95th=[ 1549],
     | 99.99th=[ 6194]
   bw (  KiB/s): min=336960, max=938088, per=100.00%, avg=884417.49, stdev=91813.69, samples=59
   iops        : min=84240, max=234522, avg=221104.37, stdev=22953.42, samples=59
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.12%, 20=15.97%, 50=32.06%
  lat (usec)   : 100=27.62%, 250=23.70%, 500=0.42%, 750=0.02%, 1000=0.01%
  lat (msec)   : 2=0.05%, 4=0.01%, 10=0.03%, 20=0.01%
  cpu          : usr=16.68%, sys=51.52%, ctx=1899320, majf=0, minf=15
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=6555326,6551874,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=864MiB/s (906MB/s), 864MiB/s-864MiB/s (906MB/s-906MB/s), io=25.0GiB (26.8GB), run=29642-29642msec
  WRITE: bw=863MiB/s (905MB/s), 863MiB/s-863MiB/s (905MB/s-905MB/s), io=25.0GiB (26.8GB), run=29642-29642msec

Disk stats (read/write):
  nvme0n1: ios=6510067/6508392, merge=0/1788, ticks=663996/117790, in_queue=782210, util=99.71%