# HPC Storage Benchmark for Ling and Grigorios

## Conclusions - text suggested for paper

> These algorithm benchmarks were collected on the CREATE HPC cluster at King's College London using the scratch storage filesystem. This filesystem uses the Ceph distributed storage system across a large number of storage nodes with diverse disk types and properties.
>
> The CREATE HPC scratch storage in file access benchmarks was found to have a throughput of 645MiB/s sequential read (1179MiB/s for cached files) and a random read latency of 37.8 ms (sdev 51.4 ms) or 12.9 ms (sdev 39.1 ms) for cached files.
>
> Sequential write throughput was 494MiB/s with a random write latency of 48.1 ms (sdev 86.4 ms).
>
> These storage benchmarks were performed during a period of low usage and were found to be subject to occasional degradation due to factors out of our control within the HPC cluster and its Ceph storage cluster.

## Notes

Results of tests were surprisingly consistent, though affected by occasional degradations in performance, presumably due to Ceph doing some background task.

How to acknowledge CREATE in papers:
https://docs.er.kcl.ac.uk/CREATE/acknowledging/

## Tests

Fio jobfiles mostly as presented at https://docs.oracle.com/en-us/iaas/Content/Block/References/samplefiocommandslinux.htm

### Case 1 - sequential read

Jobfile

[global]
bs=256K
iodepth=64
direct=1
ioengine=libaio
group_reporting
time_based
runtime=120
numjobs=4
name=raw-read
rw=read
size=1G
							
[job1]
filename=/scratch/users/k2256813/benchmark/test1

Result 1 - Saturday 8th Feb, lunchtime

job1: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
job1: Laying out IO file (1 file / 1024MiB)
Jobs: 4 (f=4): [R(4)][100.0%][r=1179MiB/s][r=4716 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=4): err= 0: pid=63081: Sat Feb  8 13:38:22 2025
  read: IOPS=4553, BW=1138MiB/s (1194MB/s)(134GiB/120082msec)
    slat (usec): min=51, max=681009, avg=191.42, stdev=1836.70
    clat (usec): min=1722, max=1657.6k, avg=56007.98, stdev=67410.10
     lat (usec): min=1912, max=1657.8k, avg=56199.64, stdev=67428.38
    clat percentiles (msec):
     |  1.00th=[    6],  5.00th=[    8], 10.00th=[    9], 20.00th=[   13],
     | 30.00th=[   18], 40.00th=[   25], 50.00th=[   34], 60.00th=[   45],
     | 70.00th=[   62], 80.00th=[   87], 90.00th=[  129], 95.00th=[  174],
     | 99.00th=[  321], 99.50th=[  388], 99.90th=[  709], 99.95th=[  827],
     | 99.99th=[  927]
   bw (  MiB/s): min=   41, max= 1524, per=100.00%, avg=1138.97, stdev=50.96, samples=960
   iops        : min=  166, max= 6096, avg=4555.70, stdev=203.84, samples=960
  lat (msec)   : 2=0.01%, 4=0.21%, 10=12.49%, 20=21.22%, 50=29.41%
  lat (msec)   : 100=20.72%, 250=13.94%, 500=1.81%, 750=0.11%, 1000=0.07%
  lat (msec)   : 2000=0.01%
  cpu          : usr=0.81%, sys=22.25%, ctx=449921, majf=0, minf=50116
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=546846,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1138MiB/s (1194MB/s), 1138MiB/s-1138MiB/s (1194MB/s-1194MB/s), io=134GiB (143GB), run=120082-120082msec

Result 2 - Sunday 9th Feb, lunchtime

job1: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
Jobs: 4 (f=4): [R(4)][100.0%][r=1178MiB/s][r=4712 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=4): err= 0: pid=234023: Sun Feb  9 12:41:20 2025
  read: IOPS=4333, BW=1083MiB/s (1136MB/s)(127GiB/120201msec)
    slat (usec): min=52, max=245760, avg=226.45, stdev=2547.87
    clat (usec): min=1872, max=5552.0k, avg=58802.42, stdev=127502.23
     lat (msec): min=2, max=5552, avg=59.03, stdev=128.28
    clat percentiles (msec):
     |  1.00th=[    6],  5.00th=[    8], 10.00th=[    9], 20.00th=[   13],
     | 30.00th=[   17], 40.00th=[   23], 50.00th=[   32], 60.00th=[   44],
     | 70.00th=[   61], 80.00th=[   86], 90.00th=[  129], 95.00th=[  178],
     | 99.00th=[  355], 99.50th=[  477], 99.90th=[ 1133], 99.95th=[ 2601],
     | 99.99th=[ 5201]
   bw (  MiB/s): min=    6, max= 1684, per=100.00%, avg=1085.00, stdev=79.24, samples=959
   iops        : min=   24, max= 6736, avg=4339.99, stdev=316.95, samples=959
  lat (msec)   : 2=0.01%, 4=0.16%, 10=13.44%, 20=22.32%, 50=28.50%
  lat (msec)   : 100=19.75%, 250=13.56%, 500=1.83%, 750=0.21%, 1000=0.11%
  lat (msec)   : 2000=0.07%, >=2000=0.06%
  cpu          : usr=0.77%, sys=21.26%, ctx=427902, majf=0, minf=59291
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=520918,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1083MiB/s (1136MB/s), 1083MiB/s-1083MiB/s (1136MB/s-1136MB/s), io=127GiB (137GB), run=120201-120201msec

### Case 2 - sequential read, 1M block size

Jobfile

[global]
bs=1M
iodepth=64
direct=1
ioengine=libaio
group_reporting
time_based
runtime=120
numjobs=4
name=raw-read
rw=read
size=1G
							
[job1]
filename=/scratch/users/k2256813/benchmark/test1

Output 1 - Sunday lunchtime

job1: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
Jobs: 4 (f=4): [R(4)][100.0%][r=1175MiB/s][r=1175 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=4): err= 0: pid=234155: Sun Feb  9 12:47:23 2025
  read: IOPS=1147, BW=1148MiB/s (1204MB/s)(135GiB/120232msec)
    slat (usec): min=85, max=5262, avg=227.18, stdev=77.99
    clat (msec): min=4, max=8281, avg=222.71, stdev=594.91
     lat (msec): min=5, max=8282, avg=222.94, stdev=594.91
    clat percentiles (msec):
     |  1.00th=[   11],  5.00th=[   18], 10.00th=[   22], 20.00th=[   32],
     | 30.00th=[   44], 40.00th=[   58], 50.00th=[   83], 60.00th=[  123],
     | 70.00th=[  184], 80.00th=[  271], 90.00th=[  430], 95.00th=[  625],
     | 99.00th=[ 2937], 99.50th=[ 5873], 99.90th=[ 7013], 99.95th=[ 7215],
     | 99.99th=[ 8020]
   bw (  MiB/s): min=  690, max= 1702, per=100.00%, avg=1148.01, stdev=49.84, samples=960
   iops        : min=  690, max= 1702, avg=1148.01, stdev=49.84, samples=960
  lat (msec)   : 10=0.76%, 20=7.06%, 50=27.30%, 100=20.13%, 250=22.73%
  lat (msec)   : 500=14.43%, 750=3.94%, 1000=1.34%, 2000=1.01%, >=2000=1.30%
  cpu          : usr=0.20%, sys=6.89%, ctx=130615, majf=0, minf=98476
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=138013,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1148MiB/s (1204MB/s), 1148MiB/s-1148MiB/s (1204MB/s-1204MB/s), io=135GiB (145GB), run=120232-120232msec

With a new file, to see if it gets put on a different disk

job1: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
job1: Laying out IO file (1 file / 1024MiB)
Jobs: 4 (f=4): [R(4)][100.0%][r=1176MiB/s][r=1176 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=4): err= 0: pid=234249: Sun Feb  9 12:51:00 2025
  read: IOPS=1162, BW=1163MiB/s (1219MB/s)(137GiB/120312msec)
    slat (usec): min=86, max=21862, avg=228.24, stdev=104.83
    clat (msec): min=5, max=10454, avg=219.87, stdev=849.33
     lat (msec): min=5, max=10454, avg=220.10, stdev=849.32
    clat percentiles (msec):
     |  1.00th=[   12],  5.00th=[   16], 10.00th=[   20], 20.00th=[   28],
     | 30.00th=[   37], 40.00th=[   51], 50.00th=[   71], 60.00th=[  105],
     | 70.00th=[  150], 80.00th=[  218], 90.00th=[  342], 95.00th=[  493],
     | 99.00th=[ 3842], 99.50th=[ 8423], 99.90th=[ 9866], 99.95th=[10000],
     | 99.99th=[10402]
   bw (  MiB/s): min=  376, max= 1708, per=100.00%, avg=1163.74, stdev=54.61, samples=960
   iops        : min=  376, max= 1708, avg=1163.49, stdev=54.60, samples=960
  lat (msec)   : 10=0.63%, 20=10.23%, 50=29.09%, 100=18.99%, 250=24.73%
  lat (msec)   : 500=11.50%, 750=2.38%, 1000=0.72%, 2000=0.60%, >=2000=1.13%
  cpu          : usr=0.21%, sys=7.00%, ctx=131824, majf=0, minf=92819
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=139874,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1163MiB/s (1219MB/s), 1163MiB/s-1163MiB/s (1219MB/s-1219MB/s), io=137GiB (147GB), run=120312-120312msec

### Case 3 - two nodes at the same time, case 1 and 2 on different files

The results below show that the network connection at the node is the limiting factor for bandwidth

job1: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
Jobs: 1 (f=1): [_(3),R(1)][100.0%][r=182MiB/s][r=728 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=4): err= 0: pid=132630: Sun Feb  9 12:53:45 2025
  read: IOPS=4664, BW=1166MiB/s (1223MB/s)(138GiB/121012msec)
    slat (usec): min=43, max=6016, avg=162.53, stdev=41.76
    clat (usec): min=1702, max=1575.0k, avg=54456.90, stdev=59482.66
     lat (usec): min=1858, max=1575.1k, avg=54619.78, stdev=59481.72
    clat percentiles (msec):
     |  1.00th=[    6],  5.00th=[    8], 10.00th=[   11], 20.00th=[   16],
     | 30.00th=[   23], 40.00th=[   31], 50.00th=[   40], 60.00th=[   50],
     | 70.00th=[   63], 80.00th=[   82], 90.00th=[  113], 95.00th=[  148],
     | 99.00th=[  259], 99.50th=[  326], 99.90th=[  701], 99.95th=[ 1020],
     | 99.99th=[ 1217]
   bw (  MiB/s): min=  810, max= 1468, per=100.00%, avg=1175.75, stdev=27.90, samples=959
   iops        : min= 3242, max= 5874, avg=4702.99, stdev=111.62, samples=959
  lat (msec)   : 2=0.01%, 4=0.19%, 10=8.97%, 20=17.08%, 50=34.36%
  lat (msec)   : 100=26.19%, 250=12.10%, 500=0.95%, 750=0.07%, 1000=0.03%
  lat (msec)   : 2000=0.05%
  cpu          : usr=0.85%, sys=20.14%, ctx=466442, majf=0, minf=54023
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=564452,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1166MiB/s (1223MB/s), 1166MiB/s-1166MiB/s (1223MB/s-1223MB/s), io=138GiB (148GB), run=121012-121012msec

job1: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
Jobs: 4 (f=4): [R(4)][100.0%][r=1182MiB/s][r=1182 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=4): err= 0: pid=234303: Sun Feb  9 12:53:46 2025
  read: IOPS=1177, BW=1177MiB/s (1234MB/s)(138GiB/120282msec)
    slat (usec): min=88, max=5111, avg=226.57, stdev=71.50
    clat (msec): min=6, max=9929, avg=217.17, stdev=805.87
     lat (msec): min=6, max=9929, avg=217.40, stdev=805.87
    clat percentiles (msec):
     |  1.00th=[   14],  5.00th=[   17], 10.00th=[   21], 20.00th=[   29],
     | 30.00th=[   38], 40.00th=[   51], 50.00th=[   71], 60.00th=[  104],
     | 70.00th=[  153], 80.00th=[  220], 90.00th=[  347], 95.00th=[  498],
     | 99.00th=[ 5537], 99.50th=[ 8087], 99.90th=[ 9060], 99.95th=[ 9194],
     | 99.99th=[ 9463]
   bw (  MiB/s): min=  702, max= 1790, per=100.00%, avg=1177.77, stdev=51.14, samples=960
   iops        : min=  702, max= 1790, avg=1177.77, stdev=51.14, samples=960
  lat (msec)   : 10=0.12%, 20=9.43%, 50=30.04%, 100=19.59%, 250=24.10%
  lat (msec)   : 500=11.75%, 750=2.59%, 1000=0.78%, 2000=0.47%, >=2000=1.13%
  cpu          : usr=0.22%, sys=7.02%, ctx=133487, majf=0, minf=82036
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=141584,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1177MiB/s (1234MB/s), 1177MiB/s-1177MiB/s (1234MB/s-1234MB/s), io=138GiB (148GB), run=120282-120282msec

### Case 4 - two nodes case 2 on same file

job1: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
Jobs: 4 (f=4): [R(4)][100.0%][r=1175MiB/s][r=1175 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=4): err= 0: pid=234371: Sun Feb  9 12:56:33 2025
  read: IOPS=1177, BW=1178MiB/s (1235MB/s)(138GiB/120238msec)
    slat (usec): min=68, max=4341, avg=227.49, stdev=111.04
    clat (msec): min=6, max=9032, avg=217.09, stdev=744.27
     lat (msec): min=6, max=9032, avg=217.32, stdev=744.27
    clat percentiles (msec):
     |  1.00th=[   13],  5.00th=[   17], 10.00th=[   20], 20.00th=[   27],
     | 30.00th=[   36], 40.00th=[   49], 50.00th=[   68], 60.00th=[  101],
     | 70.00th=[  148], 80.00th=[  218], 90.00th=[  342], 95.00th=[  514],
     | 99.00th=[ 5604], 99.50th=[ 7013], 99.90th=[ 8288], 99.95th=[ 8423],
     | 99.99th=[ 8792]
   bw (  MiB/s): min=  636, max= 1924, per=100.00%, avg=1177.84, stdev=51.23, samples=960
   iops        : min=  636, max= 1924, avg=1177.84, stdev=51.23, samples=960
  lat (msec)   : 10=0.12%, 20=10.82%, 50=30.09%, 100=18.94%, 250=23.61%
  lat (msec)   : 500=11.18%, 750=2.42%, 1000=0.68%, 2000=0.60%, >=2000=1.55%
  cpu          : usr=0.22%, sys=7.05%, ctx=133445, majf=0, minf=196978
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=141593,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1178MiB/s (1235MB/s), 1178MiB/s-1178MiB/s (1235MB/s-1235MB/s), io=138GiB (148GB), run=120238-120238msec

job1: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
Jobs: 4 (f=4): [R(4)][100.0%][r=1183MiB/s][r=1183 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=4): err= 0: pid=132689: Sun Feb  9 12:56:32 2025
  read: IOPS=1117, BW=1118MiB/s (1172MB/s)(131GiB/120339msec)
    slat (usec): min=54, max=195553, avg=421.97, stdev=4088.77
    clat (msec): min=7, max=11231, avg=228.48, stdev=732.36
     lat (msec): min=7, max=11232, avg=228.90, stdev=732.80
    clat percentiles (msec):
     |  1.00th=[   14],  5.00th=[   20], 10.00th=[   26], 20.00th=[   38],
     | 30.00th=[   52], 40.00th=[   71], 50.00th=[   95], 60.00th=[  129],
     | 70.00th=[  174], 80.00th=[  239], 90.00th=[  368], 95.00th=[  523],
     | 99.00th=[ 3641], 99.50th=[ 7215], 99.90th=[ 8356], 99.95th=[ 9060],
     | 99.99th=[10805]
   bw (  MiB/s): min=   52, max= 1770, per=100.00%, avg=1119.05, stdev=72.44, samples=960
   iops        : min=   52, max= 1770, avg=1119.05, stdev=72.44, samples=960
  lat (msec)   : 10=0.07%, 20=5.59%, 50=23.27%, 100=22.69%, 250=29.85%
  lat (msec)   : 500=13.05%, 750=2.73%, 1000=0.81%, 2000=0.61%, >=2000=1.33%
  cpu          : usr=0.30%, sys=6.92%, ctx=125102, majf=0, minf=232459
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=134538,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1118MiB/s (1172MB/s), 1118MiB/s-1118MiB/s (1172MB/s-1172MB/s), io=131GiB (141GB), run=120339-120339msec

### Case 5 - seq read 28 processes

Jobfile

[global]
bs=256K
iodepth=64
direct=1
ioengine=libaio
group_reporting
time_based
runtime=120
numjobs=28
name=raw-read
rw=read
size=1G
							
[job1]
filename=/scratch/users/k2256813/benchmark/test1

Output

job1: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 28 processes
Jobs: 28 (f=28): [R(28)][100.0%][r=1179MiB/s][r=4716 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=28): err= 0: pid=234528: Sun Feb  9 13:00:30 2025
  read: IOPS=4706, BW=1177MiB/s (1234MB/s)(138GiB/120485msec)
    slat (usec): min=54, max=5704, avg=181.58, stdev=51.82
    clat (msec): min=2, max=13460, avg=380.00, stdev=1282.10
     lat (msec): min=2, max=13460, avg=380.19, stdev=1282.10
    clat percentiles (msec):
     |  1.00th=[    5],  5.00th=[    7], 10.00th=[    9], 20.00th=[   14],
     | 30.00th=[   21], 40.00th=[   31], 50.00th=[   48], 60.00th=[   80],
     | 70.00th=[  136], 80.00th=[  243], 90.00th=[  558], 95.00th=[ 1720],
     | 99.00th=[ 7752], 99.50th=[10000], 99.90th=[12550], 99.95th=[12818],
     | 99.99th=[13221]
   bw (  MiB/s): min=   20, max= 4566, per=100.00%, avg=1362.19, stdev=33.55, samples=5809
   iops        : min=   80, max=18265, avg=5448.75, stdev=134.19, samples=5809
  lat (msec)   : 4=0.25%, 10=13.40%, 20=16.10%, 50=21.03%, 100=13.63%
  lat (msec)   : 250=16.04%, 500=8.66%, 750=2.80%, 1000=1.35%, 2000=2.08%
  lat (msec)   : >=2000=4.65%
  cpu          : usr=0.13%, sys=3.26%, ctx=544521, majf=0, minf=230030
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.7%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=567111,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1177MiB/s (1234MB/s), 1177MiB/s-1177MiB/s (1234MB/s-1234MB/s), io=138GiB (149GB), run=120485-120485msec

### Case 6 - seq read 1 process

This case shows that we can't saturate the bandwidth with 1 fio process

Input

[global]
bs=256K
iodepth=64
direct=1
ioengine=libaio
group_reporting
time_based
runtime=120
numjobs=1
name=raw-read
rw=read
size=1G
							
[job1]
filename=/scratch/users/k2256813/benchmark/test1

Output

job1: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=760MiB/s][r=3039 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=1): err= 0: pid=234628: Sun Feb  9 13:03:17 2025
  read: IOPS=3386, BW=847MiB/s (888MB/s)(99.4GiB/120211msec)
    slat (usec): min=51, max=4843, avg=165.12, stdev=124.09
    clat (usec): min=1823, max=1115.6k, avg=18728.88, stdev=36821.54
     lat (usec): min=1993, max=1115.7k, avg=18894.20, stdev=36813.90
    clat percentiles (msec):
     |  1.00th=[    4],  5.00th=[    5], 10.00th=[    6], 20.00th=[    7],
     | 30.00th=[    8], 40.00th=[    9], 50.00th=[   10], 60.00th=[   11],
     | 70.00th=[   13], 80.00th=[   18], 90.00th=[   36], 95.00th=[   68],
     | 99.00th=[  163], 99.50th=[  222], 99.90th=[  477], 99.95th=[  617],
     | 99.99th=[  827]
   bw (  KiB/s): min=562688, max=1082880, per=100.00%, avg=868390.40, stdev=99229.00, samples=240
   iops        : min= 2198, max= 4230, avg=3392.15, stdev=387.61, samples=240
  lat (msec)   : 2=0.01%, 4=1.98%, 10=52.14%, 20=28.96%, 50=9.76%
  lat (msec)   : 100=4.44%, 250=2.31%, 500=0.31%, 750=0.07%, 1000=0.02%
  lat (msec)   : 2000=0.01%
  cpu          : usr=1.65%, sys=55.86%, ctx=233397, majf=0, minf=17998
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=407121,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=847MiB/s (888MB/s), 847MiB/s-847MiB/s (888MB/s-888MB/s), io=99.4GiB (107GB), run=120211-120211msec

With 2 output - around 13:05

job1: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 2 processes
Jobs: 1 (f=1): [R(1),_(1)][100.0%][eta 00m:00s]                        
job1: (groupid=0, jobs=2): err= 0: pid=132833: Sun Feb  9 13:05:21 2025
  read: IOPS=2780, BW=695MiB/s (729MB/s)(85.6GiB/126138msec)
    slat (usec): min=38, max=7102, avg=160.43, stdev=93.72
    clat (usec): min=655, max=15486k, avg=45376.51, stdev=546472.40
     lat (usec): min=1604, max=15486k, avg=45537.18, stdev=546470.37
    clat percentiles (msec):
     |  1.00th=[    3],  5.00th=[    4], 10.00th=[    5], 20.00th=[    6],
     | 30.00th=[    6], 40.00th=[    7], 50.00th=[    9], 60.00th=[   11],
     | 70.00th=[   15], 80.00th=[   23], 90.00th=[   43], 95.00th=[   74],
     | 99.00th=[  184], 99.50th=[  292], 99.90th=[12953], 99.95th=[13758],
     | 99.99th=[14563]
   bw (  KiB/s): min=26624, max=1438208, per=100.00%, avg=765534.15, stdev=176710.53, samples=469
   iops        : min=  104, max= 5618, avg=2990.36, stdev=690.27, samples=469
  lat (usec)   : 750=0.01%
  lat (msec)   : 2=0.18%, 4=9.27%, 10=48.65%, 20=19.58%, 50=13.97%
  lat (msec)   : 100=5.34%, 250=2.40%, 500=0.19%, 750=0.02%, 1000=0.05%
  lat (msec)   : 2000=0.10%, >=2000=0.25%
  cpu          : usr=0.64%, sys=23.28%, ctx=262203, majf=0, minf=24740
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=350722,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=695MiB/s (729MB/s), 695MiB/s-695MiB/s (729MB/s-729MB/s), io=85.6GiB (91.9GB), run=126138-126138msec

Much slower than expected - why?

With 3 output - around 13:05

job1: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 3 processes
Jobs: 1 (f=1): [_(2),R(1)][100.0%][r=15.8MiB/s][r=63 IOPS][eta 00m:00s]     
job1: (groupid=0, jobs=3): err= 0: pid=234673: Sun Feb  9 13:05:46 2025
  read: IOPS=3375, BW=844MiB/s (885MB/s)(102GiB/124242msec)
    slat (usec): min=46, max=14479, avg=223.24, stdev=139.23
    clat (usec): min=1650, max=15990k, avg=56248.34, stdev=614371.38
     lat (usec): min=1810, max=15990k, avg=56471.82, stdev=614367.78
    clat percentiles (msec):
     |  1.00th=[    3],  5.00th=[    5], 10.00th=[    5], 20.00th=[    7],
     | 30.00th=[    8], 40.00th=[    9], 50.00th=[   11], 60.00th=[   13],
     | 70.00th=[   18], 80.00th=[   27], 90.00th=[   52], 95.00th=[   83],
     | 99.00th=[  188], 99.50th=[  305], 99.90th=[13087], 99.95th=[13758],
     | 99.99th=[14966]
   bw (  KiB/s): min=86528, max=1634304, per=100.00%, avg=901704.97, stdev=104746.15, samples=714
   iops        : min=  338, max= 6384, avg=3522.29, stdev=409.16, samples=714
  lat (msec)   : 2=0.06%, 4=4.43%, 10=45.39%, 20=23.98%, 50=15.91%
  lat (msec)   : 100=6.75%, 250=2.85%, 500=0.21%, 750=0.02%, 1000=0.01%
  lat (msec)   : 2000=0.01%, >=2000=0.39%
  cpu          : usr=0.76%, sys=25.90%, ctx=313501, majf=0, minf=31555
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=419344,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=844MiB/s (885MB/s), 844MiB/s-844MiB/s (885MB/s-885MB/s), io=102GiB (110GB), run=124242-124242msec

Much slower than expected - why?

Repeat of 2

job1: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 2 processes
Jobs: 2 (f=2): [R(2)][100.0%][r=1172MiB/s][r=4688 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=2): err= 0: pid=132885: Sun Feb  9 13:08:13 2025
  read: IOPS=4611, BW=1153MiB/s (1209MB/s)(136GiB/120427msec)
    slat (usec): min=44, max=8028, avg=143.60, stdev=47.43
    clat (usec): min=1587, max=2149.6k, avg=27564.93, stdev=54307.82
     lat (usec): min=1756, max=2149.7k, avg=27708.79, stdev=54306.66
    clat percentiles (msec):
     |  1.00th=[    5],  5.00th=[    6], 10.00th=[    7], 20.00th=[    9],
     | 30.00th=[   11], 40.00th=[   13], 50.00th=[   16], 60.00th=[   20],
     | 70.00th=[   26], 80.00th=[   36], 90.00th=[   56], 95.00th=[   82],
     | 99.00th=[  163], 99.50th=[  226], 99.90th=[  827], 99.95th=[ 1301],
     | 99.99th=[ 1703]
   bw (  MiB/s): min=  538, max= 1485, per=100.00%, avg=1157.16, stdev=62.41, samples=479
   iops        : min= 2152, max= 5940, avg=4628.67, stdev=249.65, samples=479
  lat (msec)   : 2=0.01%, 4=0.94%, 10=27.63%, 20=31.60%, 50=28.04%
  lat (msec)   : 100=8.42%, 250=2.96%, 500=0.25%, 750=0.04%, 1000=0.04%
  lat (msec)   : 2000=0.07%, >=2000=0.01%
  cpu          : usr=1.09%, sys=34.48%, ctx=400565, majf=0, minf=52028
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=555316,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1153MiB/s (1209MB/s), 1153MiB/s-1153MiB/s (1209MB/s-1209MB/s), io=136GiB (146GB), run=120427-120427msec

Repeat of 3

job1: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 3 processes
Jobs: 3 (f=3): [R(3)][100.0%][r=1178MiB/s][r=4710 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=3): err= 0: pid=234725: Sun Feb  9 13:08:05 2025
  read: IOPS=4675, BW=1169MiB/s (1226MB/s)(137GiB/120073msec)
    slat (usec): min=50, max=14698, avg=181.83, stdev=67.89
    clat (usec): min=1839, max=2187.6k, avg=40870.42, stdev=63044.02
     lat (msec): min=2, max=2187, avg=41.05, stdev=63.04
    clat percentiles (msec):
     |  1.00th=[    6],  5.00th=[    7], 10.00th=[    9], 20.00th=[   12],
     | 30.00th=[   16], 40.00th=[   21], 50.00th=[   27], 60.00th=[   34],
     | 70.00th=[   44], 80.00th=[   59], 90.00th=[   86], 95.00th=[  115],
     | 99.00th=[  203], 99.50th=[  262], 99.90th=[  902], 99.95th=[ 1435],
     | 99.99th=[ 2022]
   bw (  MiB/s): min=  694, max= 1596, per=100.00%, avg=1169.26, stdev=44.76, samples=720
   iops        : min= 2776, max= 6386, avg=4677.03, stdev=179.04, samples=720
  lat (msec)   : 2=0.01%, 4=0.17%, 10=14.21%, 20=25.23%, 50=35.24%
  lat (msec)   : 100=18.10%, 250=6.49%, 500=0.38%, 750=0.06%, 1000=0.03%
  lat (msec)   : 2000=0.08%, >=2000=0.01%
  cpu          : usr=1.02%, sys=29.60%, ctx=433163, majf=0, minf=35168
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=561432,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1169MiB/s (1226MB/s), 1169MiB/s-1169MiB/s (1226MB/s-1226MB/s), io=137GiB (147GB), run=120073-120073msec

Repeat of 1 - again doesn't saturate, seems to be genuine

job1: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=813MiB/s][r=3252 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=1): err= 0: pid=234793: Sun Feb  9 13:10:38 2025
  read: IOPS=3327, BW=832MiB/s (872MB/s)(97.6GiB/120131msec)
    slat (usec): min=51, max=4484, avg=167.07, stdev=126.14
    clat (usec): min=1378, max=1915.4k, avg=19060.43, stdev=51280.28
     lat (usec): min=1831, max=1915.5k, avg=19227.67, stdev=51273.92
    clat percentiles (msec):
     |  1.00th=[    4],  5.00th=[    5], 10.00th=[    6], 20.00th=[    7],
     | 30.00th=[    8], 40.00th=[    9], 50.00th=[   10], 60.00th=[   11],
     | 70.00th=[   13], 80.00th=[   17], 90.00th=[   34], 95.00th=[   65],
     | 99.00th=[  163], 99.50th=[  224], 99.90th=[  818], 99.95th=[ 1053],
     | 99.99th=[ 1519]
   bw (  KiB/s): min=466944, max=1054208, per=100.00%, avg=852744.53, stdev=110314.08, samples=240
   iops        : min= 1824, max= 4118, avg=3331.03, stdev=430.91, samples=240
  lat (msec)   : 2=0.02%, 4=2.54%, 10=52.92%, 20=28.42%, 50=9.31%
  lat (msec)   : 100=4.28%, 250=2.10%, 500=0.20%, 750=0.08%, 1000=0.06%
  lat (msec)   : 2000=0.06%
  cpu          : usr=1.67%, sys=55.49%, ctx=226708, majf=0, minf=4111
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=399787,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=832MiB/s (872MB/s), 832MiB/s-832MiB/s (872MB/s-872MB/s), io=97.6GiB (105GB), run=120131-120131msec

### Case 7 - random read latency

Jobfile

[global]
bs=4K
iodepth=1
direct=1
ioengine=libaio
group_reporting
time_based
runtime=120
numjobs=1
name=readlatency-test-job
rw=randread
							
[job1]
filename=/scratch/users/k2256813/benchmark/test1

Output

job1: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=284KiB/s][r=71 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=1): err= 0: pid=234847: Sun Feb  9 13:14:02 2025
  read: IOPS=76, BW=306KiB/s (314kB/s)(35.9MiB/120002msec)
    slat (usec): min=127, max=500, avg=169.64, stdev=17.16
    clat (usec): min=731, max=613984, avg=12885.33, stdev=39086.79
     lat (usec): min=881, max=614146, avg=13055.19, stdev=39087.48
    clat percentiles (usec):
     |  1.00th=[  1020],  5.00th=[  1123], 10.00th=[  1188], 20.00th=[  1287],
     | 30.00th=[  1369], 40.00th=[  1532], 50.00th=[  1729], 60.00th=[  1975],
     | 70.00th=[  2311], 80.00th=[  2933], 90.00th=[ 29492], 95.00th=[ 84411],
     | 99.00th=[196084], 99.50th=[248513], 99.90th=[408945], 99.95th=[476054],
     | 99.99th=[616563]
   bw (  KiB/s): min=    8, max=  880, per=100.00%, avg=307.04, stdev=148.48, samples=239
   iops        : min=    2, max=  220, avg=76.76, stdev=37.12, samples=239
  lat (usec)   : 750=0.02%, 1000=0.77%
  lat (msec)   : 2=60.25%, 4=21.36%, 10=2.94%, 20=2.84%, 50=4.23%
  lat (msec)   : 100=3.71%, 250=3.38%, 500=0.47%, 750=0.02%
  cpu          : usr=0.07%, sys=1.36%, ctx=9221, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=9190,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=306KiB/s (314kB/s), 306KiB/s-306KiB/s (314kB/s-314kB/s), io=35.9MiB (37.6MB), run=120002-120002msec

### Case 8 - random read bandwidth

Jobfile

[global]
bs=256K
iodepth=64
direct=1
ioengine=libaio
group_reporting
time_based
runtime=120
numjobs=4
name=raw-randread
rw=randread
							
[job1]
filename=/scratch/users/k2256813/benchmark/test1

Output

job1: (g=0): rw=randread, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
Jobs: 2 (f=2): [r(1),_(1),r(1),_(1)][100.0%][r=167MiB/s][r=668 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=4): err= 0: pid=133014: Sun Feb  9 13:18:52 2025
  read: IOPS=4657, BW=1164MiB/s (1221MB/s)(138GiB/120960msec)
    slat (usec): min=59, max=8884, avg=174.61, stdev=33.64
    clat (msec): min=2, max=1956, avg=54.69, stdev=106.58
     lat (msec): min=2, max=1956, avg=54.87, stdev=106.58
    clat percentiles (msec):
     |  1.00th=[    6],  5.00th=[    7], 10.00th=[    8], 20.00th=[   10],
     | 30.00th=[   11], 40.00th=[   14], 50.00th=[   17], 60.00th=[   24],
     | 70.00th=[   37], 80.00th=[   65], 90.00th=[  142], 95.00th=[  239],
     | 99.00th=[  531], 99.50th=[  684], 99.90th=[ 1099], 99.95th=[ 1318],
     | 99.99th=[ 1620]
   bw (  MiB/s): min=  738, max= 1467, per=100.00%, avg=1173.21, stdev=26.20, samples=960
   iops        : min= 2954, max= 5868, avg=4692.84, stdev=104.79, samples=960
  lat (msec)   : 4=0.16%, 10=25.23%, 20=29.85%, 50=20.68%, 100=10.26%
  lat (msec)   : 250=9.30%, 500=3.37%, 750=0.76%, 1000=0.24%, 2000=0.15%
  cpu          : usr=1.02%, sys=21.61%, ctx=438328, majf=0, minf=35451
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=563393,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1164MiB/s (1221MB/s), 1164MiB/s-1164MiB/s (1221MB/s-1221MB/s), io=138GiB (148GB), run=120960-120960msec

Output with 1 process

job1: (g=0): rw=randread, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=1094MiB/s][r=4375 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=1): err= 0: pid=234922: Sun Feb  9 13:20:28 2025
  read: IOPS=4210, BW=1053MiB/s (1104MB/s)(124GiB/120179msec)
    slat (usec): min=51, max=6953, avg=141.15, stdev=42.01
    clat (usec): min=1724, max=1883.1k, avg=15056.63, stdev=41548.21
     lat (usec): min=1864, max=1883.2k, avg=15197.93, stdev=41548.38
    clat percentiles (msec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    4], 20.00th=[    4],
     | 30.00th=[    5], 40.00th=[    5], 50.00th=[    6], 60.00th=[    6],
     | 70.00th=[    7], 80.00th=[   10], 90.00th=[   31], 95.00th=[   69],
     | 99.00th=[  180], 99.50th=[  230], 99.90th=[  456], 99.95th=[  651],
     | 99.99th=[ 1284]
   bw (  MiB/s): min=  764, max= 1157, per=100.00%, avg=1053.96, stdev=63.51, samples=240
   iops        : min= 3056, max= 4628, avg=4215.84, stdev=254.02, samples=240
  lat (msec)   : 2=0.03%, 4=27.40%, 10=53.84%, 20=6.18%, 50=5.57%
  lat (msec)   : 100=4.01%, 250=2.57%, 500=0.32%, 750=0.04%, 1000=0.02%
  lat (msec)   : 2000=0.02%
  cpu          : usr=2.08%, sys=59.57%, ctx=243449, majf=0, minf=19379
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=505964,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1053MiB/s (1104MB/s), 1053MiB/s-1053MiB/s (1104MB/s-1104MB/s), io=124GiB (133GB), run=120179-120179msec

### Case 9 - random rw

Input

[global]
bs=256K
iodepth=64
direct=1
ioengine=libaio
group_reporting
time_based
runtime=120
numjobs=4
name=raw-randreadwrite
rw=randrw
size=1G
							
[job1]
filename=/scratch/users/k2256813/benchmark/test3

Output

job1: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
job1: Laying out IO file (1 file / 1024MiB)
Jobs: 4 (f=4): [m(4)][100.0%][eta 00m:00s]                                           
job1: (groupid=0, jobs=4): err= 0: pid=235033: Sun Feb  9 13:26:10 2025
  read: IOPS=1158, BW=290MiB/s (304MB/s)(36.7GiB/129904msec)
    slat (usec): min=50, max=1969, avg=198.47, stdev=48.49
    clat (usec): min=1717, max=15051k, avg=92206.70, stdev=458379.24
     lat (usec): min=1899, max=15052k, avg=92405.50, stdev=458380.92
    clat percentiles (msec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    4],
     | 30.00th=[    5], 40.00th=[    6], 50.00th=[   10], 60.00th=[   20],
     | 70.00th=[   36], 80.00th=[   71], 90.00th=[  174], 95.00th=[  334],
     | 99.00th=[ 1116], 99.50th=[ 2400], 99.90th=[ 6812], 99.95th=[ 9597],
     | 99.99th=[12818]
   bw (  KiB/s): min=159744, max=479744, per=100.00%, avg=320801.74, stdev=14656.32, samples=960
   iops        : min=  624, max= 1874, avg=1253.11, stdev=57.25, samples=960
  write: IOPS=1159, BW=290MiB/s (304MB/s)(36.8GiB/129904msec); 0 zone resets
    slat (usec): min=58, max=4829, avg=239.87, stdev=109.04
    clat (msec): min=5, max=15012, avg=127.26, stdev=499.30
     lat (msec): min=5, max=15013, avg=127.50, stdev=499.30
    clat percentiles (msec):
     |  1.00th=[   10],  5.00th=[   15], 10.00th=[   18], 20.00th=[   22],
     | 30.00th=[   27], 40.00th=[   32], 50.00th=[   39], 60.00th=[   51],
     | 70.00th=[   72], 80.00th=[  114], 90.00th=[  226], 95.00th=[  401],
     | 99.00th=[ 1217], 99.50th=[ 2802], 99.90th=[ 9194], 99.95th=[ 9731],
     | 99.99th=[12818]
   bw (  KiB/s): min=169472, max=490496, per=100.00%, avg=321100.32, stdev=14166.73, samples=960
   iops        : min=  662, max= 1916, avg=1254.28, stdev=55.34, samples=960
  lat (msec)   : 2=0.07%, 4=14.69%, 10=10.89%, 20=12.91%, 50=28.83%
  lat (msec)   : 100=13.59%, 250=11.08%, 500=4.56%, 750=1.47%, 1000=0.66%
  lat (msec)   : 2000=0.66%, >=2000=0.57%
  cpu          : usr=1.23%, sys=12.33%, ctx=257127, majf=0, minf=16669
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=150473,150667,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=290MiB/s (304MB/s), 290MiB/s-290MiB/s (304MB/s-304MB/s), io=36.7GiB (39.4GB), run=129904-129904msec
  WRITE: bw=290MiB/s (304MB/s), 290MiB/s-290MiB/s (304MB/s-304MB/s), io=36.8GiB (39.5GB), run=129904-129904msec

Does this show that performance of other benchmarks shows caching?
Test with a much larger file that won't be cached

### Case 10 - 100GB file seq read

Jobfile

[global]
bs=256K
iodepth=64
direct=1
ioengine=libaio
group_reporting
time_based
runtime=60
numjobs=4
name=raw-read
rw=read
size=100G
							
[job1]
filename=/scratch/users/k2256813/benchmark/test_100G

Output

job1: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
job1: Laying out IO file (1 file / 102400MiB)
Jobs: 4 (f=4): [R(4)][100.0%][r=585MiB/s][r=2341 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=4): err= 0: pid=235789: Sun Feb  9 13:33:08 2025
  read: IOPS=2580, BW=645MiB/s (677MB/s)(38.0GiB/60326msec)
    slat (usec): min=54, max=8994, avg=227.18, stdev=188.56
    clat (usec): min=1661, max=1025.0k, avg=98924.95, stdev=136291.56
     lat (usec): min=1833, max=1025.2k, avg=99152.32, stdev=136261.29
    clat percentiles (msec):
     |  1.00th=[    4],  5.00th=[    5], 10.00th=[    7], 20.00th=[   12],
     | 30.00th=[   19], 40.00th=[   28], 50.00th=[   42], 60.00th=[   59],
     | 70.00th=[   85], 80.00th=[  171], 90.00th=[  296], 95.00th=[  401],
     | 99.00th=[  600], 99.50th=[  693], 99.90th=[  877], 99.95th=[  919],
     | 99.99th=[ 1011]
   bw (  KiB/s): min=250880, max=1191936, per=100.00%, avg=663210.67, stdev=41274.75, samples=480
   iops        : min=  980, max= 4656, avg=2590.68, stdev=161.23, samples=480
  lat (msec)   : 2=0.07%, 4=2.65%, 10=15.38%, 20=13.39%, 50=23.75%
  lat (msec)   : 100=17.65%, 250=14.05%, 500=10.69%, 750=2.09%, 1000=0.27%
  lat (msec)   : 2000=0.02%
  cpu          : usr=0.44%, sys=14.92%, ctx=124150, majf=0, minf=27026
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=155695,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=645MiB/s (677MB/s), 645MiB/s-645MiB/s (677MB/s-677MB/s), io=38.0GiB (40.8GB), run=60326-60326msec

Repeat after deleting file

job1: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
job1: Laying out IO file (1 file / 102400MiB)
Jobs: 4 (f=4): [R(4)][9.3%][r=771MiB/s][r=3085 IOPS][eta 09m:56s]  
job1: (groupid=0, jobs=4): err= 0: pid=235900: Sun Feb  9 13:36:44 2025
  read: IOPS=2531, BW=633MiB/s (664MB/s)(37.2GiB/60201msec)
    slat (usec): min=53, max=7276, avg=224.25, stdev=174.54
    clat (usec): min=1647, max=1528.2k, avg=100870.34, stdev=136508.59
     lat (usec): min=1831, max=1528.3k, avg=101094.83, stdev=136480.13
    clat percentiles (msec):
     |  1.00th=[    4],  5.00th=[    6], 10.00th=[    7], 20.00th=[   12],
     | 30.00th=[   21], 40.00th=[   32], 50.00th=[   48], 60.00th=[   69],
     | 70.00th=[  101], 80.00th=[  167], 90.00th=[  275], 95.00th=[  393],
     | 99.00th=[  609], 99.50th=[  718], 99.90th=[  978], 99.95th=[ 1385],
     | 99.99th=[ 1519]
   bw (  KiB/s): min=296960, max=1181184, per=100.00%, avg=649102.70, stdev=43387.20, samples=480
   iops        : min= 1160, max= 4614, avg=2535.53, stdev=169.47, samples=480
  lat (msec)   : 2=0.03%, 4=2.19%, 10=14.94%, 20=12.22%, 50=21.83%
  lat (msec)   : 100=18.77%, 250=18.27%, 500=9.51%, 750=1.78%, 1000=0.38%
  lat (msec)   : 2000=0.09%
  cpu          : usr=0.46%, sys=14.52%, ctx=123222, majf=0, minf=29709
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=152389,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=633MiB/s (664MB/s), 633MiB/s-633MiB/s (664MB/s-664MB/s), io=37.2GiB (39.9GB), run=60201-60201msec

Repeat keeping file

job1: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
Jobs: 4 (f=4): [R(4)][100.0%][r=658MiB/s][r=2632 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=4): err= 0: pid=235934: Sun Feb  9 13:37:53 2025
  read: IOPS=3818, BW=955MiB/s (1001MB/s)(56.2GiB/60253msec)
    slat (usec): min=52, max=14983, avg=199.21, stdev=113.11
    clat (usec): min=1652, max=1425.2k, avg=66776.98, stdev=95989.52
     lat (usec): min=1815, max=1425.4k, avg=66976.43, stdev=95978.41
    clat percentiles (msec):
     |  1.00th=[    4],  5.00th=[    7], 10.00th=[    8], 20.00th=[   12],
     | 30.00th=[   17], 40.00th=[   24], 50.00th=[   33], 60.00th=[   45],
     | 70.00th=[   64], 80.00th=[   95], 90.00th=[  161], 95.00th=[  255],
     | 99.00th=[  477], 99.50th=[  567], 99.90th=[  810], 99.95th=[  919],
     | 99.99th=[  995]
   bw (  KiB/s): min=313344, max=1880064, per=100.00%, avg=980548.27, stdev=78527.08, samples=480
   iops        : min= 1224, max= 7344, avg=3830.27, stdev=306.75, samples=480
  lat (msec)   : 2=0.01%, 4=1.11%, 10=14.62%, 20=19.72%, 50=28.10%
  lat (msec)   : 100=17.62%, 250=13.63%, 500=4.39%, 750=0.64%, 1000=0.15%
  lat (msec)   : 2000=0.01%
  cpu          : usr=0.69%, sys=19.77%, ctx=186279, majf=0, minf=51174
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=230068,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=955MiB/s (1001MB/s), 955MiB/s-955MiB/s (1001MB/s-1001MB/s), io=56.2GiB (60.3GB), run=60253-60253msec

Repeat after deleting file

job1: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
job1: Laying out IO file (1 file / 102400MiB)
Jobs: 4 (f=4): [R(4)][100.0%][r=707MiB/s][r=2829 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=4): err= 0: pid=236216: Sun Feb  9 13:47:03 2025
  read: IOPS=2707, BW=677MiB/s (710MB/s)(39.9GiB/60389msec)
    slat (usec): min=48, max=7927, avg=225.54, stdev=174.79
    clat (usec): min=1648, max=1234.9k, avg=94278.95, stdev=133825.06
     lat (usec): min=1807, max=1235.1k, avg=94504.71, stdev=133797.42
    clat percentiles (msec):
     |  1.00th=[    4],  5.00th=[    6], 10.00th=[    7], 20.00th=[   12],
     | 30.00th=[   20], 40.00th=[   29], 50.00th=[   42], 60.00th=[   57],
     | 70.00th=[   81], 80.00th=[  150], 90.00th=[  275], 95.00th=[  372],
     | 99.00th=[  634], 99.50th=[  743], 99.90th=[  969], 99.95th=[ 1028],
     | 99.99th=[ 1200]
   bw (  KiB/s): min=276791, max=1096192, per=100.00%, avg=696569.50, stdev=41634.45, samples=480
   iops        : min= 1081, max= 4282, avg=2720.97, stdev=162.64, samples=480
  lat (msec)   : 2=0.05%, 4=2.30%, 10=15.25%, 20=13.27%, 50=25.11%
  lat (msec)   : 100=18.11%, 250=14.54%, 500=9.09%, 750=1.82%, 1000=0.39%
  lat (msec)   : 2000=0.06%
  cpu          : usr=0.44%, sys=15.65%, ctx=131238, majf=0, minf=24737
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=163513,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=677MiB/s (710MB/s), 677MiB/s-677MiB/s (710MB/s-710MB/s), io=39.9GiB (42.9GB), run=60389-60389msec

### Case 11 - 100GB random read

Jobfile

[global]
bs=256K
iodepth=64
direct=1
ioengine=libaio
group_reporting
time_based
runtime=60
numjobs=4
name=raw-read
rw=randread
size=100G
							
[job1]
filename=/scratch/users/k2256813/benchmark/test_100G

Output

job1: (g=0): rw=randread, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
job1: Laying out IO file (1 file / 102400MiB)
Jobs: 4 (f=4): [r(4)][16.6%][r=1159MiB/s][r=4637 IOPS][eta 05m:06s] 
job1: (groupid=0, jobs=4): err= 0: pid=236025: Sun Feb  9 13:42:50 2025
  read: IOPS=4531, BW=1133MiB/s (1188MB/s)(66.8GiB/60391msec)
    slat (usec): min=53, max=1090, avg=189.44, stdev=27.98
    clat (msec): min=2, max=2202, avg=56.20, stdev=83.07
     lat (msec): min=2, max=2203, avg=56.39, stdev=83.07
    clat percentiles (msec):
     |  1.00th=[    6],  5.00th=[    7], 10.00th=[    9], 20.00th=[   15],
     | 30.00th=[   20], 40.00th=[   24], 50.00th=[   29], 60.00th=[   36],
     | 70.00th=[   50], 80.00th=[   73], 90.00th=[  130], 95.00th=[  220],
     | 99.00th=[  397], 99.50th=[  514], 99.90th=[  793], 99.95th=[  969],
     | 99.99th=[ 1536]
   bw (  MiB/s): min=  840, max= 1305, per=100.00%, avg=1139.16, stdev=25.40, samples=480
   iops        : min= 3360, max= 5222, avg=4556.63, stdev=101.60, samples=480
  lat (msec)   : 4=0.14%, 10=12.32%, 20=19.50%, 50=38.26%, 100=16.12%
  lat (msec)   : 250=10.28%, 500=2.84%, 750=0.42%, 1000=0.07%, 2000=0.05%
  lat (msec)   : >=2000=0.01%
  cpu          : usr=0.92%, sys=22.46%, ctx=218438, majf=0, minf=16697
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=273650,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1133MiB/s (1188MB/s), 1133MiB/s-1133MiB/s (1188MB/s-1188MB/s), io=66.8GiB (71.7GB), run=60391-60391msec

This is faster than I expected - were the previous tests slowed down?

### Case 12 - random rw latency

Input

Output

job1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [m(1)][100.0%][r=24KiB/s,w=24KiB/s][r=6,w=6 IOPS][eta 00m:00s]   
job1: (groupid=0, jobs=1): err= 0: pid=236267: Sun Feb  9 13:51:17 2025
  read: IOPS=14, BW=58.7KiB/s (60.1kB/s)(7048KiB/120075msec)
    slat (usec): min=130, max=756, avg=173.90, stdev=26.19
    clat (usec): min=891, max=417542, avg=17532.07, stdev=42407.83
     lat (usec): min=1055, max=417716, avg=17706.18, stdev=42409.15
    clat percentiles (usec):
     |  1.00th=[  1045],  5.00th=[  1156], 10.00th=[  1221], 20.00th=[  1319],
     | 30.00th=[  1467], 40.00th=[  1729], 50.00th=[  2089], 60.00th=[  2638],
     | 70.00th=[  8717], 80.00th=[ 17695], 90.00th=[ 47449], 95.00th=[ 89654],
     | 99.00th=[233833], 99.50th=[287310], 99.90th=[367002], 99.95th=[417334],
     | 99.99th=[417334]
   bw (  KiB/s): min=    8, max=  200, per=100.00%, avg=61.29, stdev=38.82, samples=230
   iops        : min=    2, max=   50, avg=15.32, stdev= 9.71, samples=230
  write: IOPS=15, BW=61.3KiB/s (62.7kB/s)(7356KiB/120075msec); 0 zone resets
    slat (usec): min=152, max=109592, avg=236.55, stdev=2551.55
    clat (msec): min=2, max=722, avg=48.09, stdev=86.36
     lat (msec): min=2, max=722, avg=48.32, stdev=86.37
    clat percentiles (msec):
     |  1.00th=[    3],  5.00th=[    4], 10.00th=[    4], 20.00th=[    5],
     | 30.00th=[    6], 40.00th=[    6], 50.00th=[    8], 60.00th=[   17],
     | 70.00th=[   29], 80.00th=[   69], 90.00th=[  163], 95.00th=[  247],
     | 99.00th=[  393], 99.50th=[  460], 99.90th=[  592], 99.95th=[  726],
     | 99.99th=[  726]
   bw (  KiB/s): min=    8, max=  176, per=99.57%, avg=61.52, stdev=34.02, samples=239
   iops        : min=    2, max=   44, avg=15.38, stdev= 8.50, samples=239
  lat (usec)   : 1000=0.14%
  lat (msec)   : 2=23.30%, 4=16.36%, 10=22.02%, 20=10.91%, 50=10.64%
  lat (msec)   : 100=6.55%, 250=7.25%, 500=2.64%, 750=0.19%
  cpu          : usr=0.02%, sys=0.56%, ctx=3603, majf=0, minf=15
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1762,1839,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=58.7KiB/s (60.1kB/s), 58.7KiB/s-58.7KiB/s (60.1kB/s-60.1kB/s), io=7048KiB (7217kB), run=120075-120075msec
  WRITE: bw=61.3KiB/s (62.7kB/s), 61.3KiB/s-61.3KiB/s (62.7kB/s-62.7kB/s), io=7356KiB (7533kB), run=120075-120075msec

### Case 13 -

Input

[global]
bs=256K
iodepth=64
direct=1
ioengine=libaio
group_reporting
time_based
runtime=120
numjobs=4
name=raw-write
rw=write
size=1G
							
[job1]
filename=/scratch/users/k2256813/benchmark/test6

Output

job1: (g=0): rw=write, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
job1: Laying out IO file (1 file / 1024MiB)
job1: Laying out IO file (1 file / 1024MiB)
job1: Laying out IO file (1 file / 1024MiB)
job1: Laying out IO file (1 file / 1024MiB)
Jobs: 4 (f=4): [W(4)][100.0%][w=507MiB/s][w=2028 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=4): err= 0: pid=134086: Sun Feb  9 13:53:25 2025
  write: IOPS=1976, BW=494MiB/s (518MB/s)(58.0GiB/120240msec); 0 zone resets
    slat (usec): min=50, max=14511, avg=228.18, stdev=388.04
    clat (msec): min=3, max=1838, avg=129.21, stdev=112.81
     lat (msec): min=3, max=1838, avg=129.44, stdev=112.81
    clat percentiles (msec):
     |  1.00th=[   20],  5.00th=[   33], 10.00th=[   41], 20.00th=[   55],
     | 30.00th=[   69], 40.00th=[   83], 50.00th=[   99], 60.00th=[  116],
     | 70.00th=[  140], 80.00th=[  178], 90.00th=[  253], 95.00th=[  334],
     | 99.00th=[  575], 99.50th=[  693], 99.90th=[ 1062], 99.95th=[ 1217],
     | 99.99th=[ 1519]
   bw (  KiB/s): min=210432, max=797184, per=100.00%, avg=506612.46, stdev=25013.38, samples=960
   iops        : min=  822, max= 3114, avg=1978.83, stdev=97.73, samples=960
  lat (msec)   : 4=0.01%, 10=0.14%, 20=1.01%, 50=15.55%, 100=34.54%
  lat (msec)   : 250=38.47%, 500=8.77%, 750=1.15%, 1000=0.25%, 2000=0.12%
  cpu          : usr=1.47%, sys=9.46%, ctx=64617, majf=0, minf=39458
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,237694,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=494MiB/s (518MB/s), 494MiB/s-494MiB/s (518MB/s-518MB/s), io=58.0GiB (62.3GB), run=120240-120240msec

### Case 14 - random read latency on 100G file

Input

[global]
bs=4K
iodepth=1
direct=1
ioengine=libaio
group_reporting
time_based
runtime=120
numjobs=1
name=readlatency-test-job
rw=randread
size=100G
							
[job1]
filename=/scratch/users/k2256813/benchmark/test_100G

Output

job1: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.28
Starting 1 process
job1: Laying out IO file (1 file / 102400MiB)
Jobs: 1 (f=1): [r(1)][100.0%][r=148KiB/s][r=37 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=1): err= 0: pid=236406: Sun Feb  9 13:59:13 2025
  read: IOPS=26, BW=105KiB/s (108kB/s)(12.3MiB/120086msec)
    slat (usec): min=107, max=325, avg=178.93, stdev=27.32
    clat (usec): min=1709, max=869548, avg=37807.73, stdev=51401.37
     lat (usec): min=1871, max=869718, avg=37986.95, stdev=51402.18
    clat percentiles (msec):
     |  1.00th=[    7],  5.00th=[   12], 10.00th=[   14], 20.00th=[   16],
     | 30.00th=[   18], 40.00th=[   20], 50.00th=[   21], 60.00th=[   23],
     | 70.00th=[   29], 80.00th=[   43], 90.00th=[   81], 95.00th=[  128],
     | 99.00th=[  257], 99.50th=[  338], 99.90th=[  468], 99.95th=[  709],
     | 99.99th=[  869]
   bw (  KiB/s): min=    8, max=  200, per=100.00%, avg=106.22, stdev=35.73, samples=238
   iops        : min=    2, max=   50, avg=26.55, stdev= 8.93, samples=238
  lat (msec)   : 2=0.13%, 4=0.54%, 10=2.21%, 20=43.09%, 50=37.17%
  lat (msec)   : 100=9.36%, 250=6.42%, 500=0.98%, 750=0.06%, 1000=0.03%
  cpu          : usr=0.02%, sys=0.50%, ctx=3161, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3161,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=105KiB/s (108kB/s), 105KiB/s-105KiB/s (108kB/s-108kB/s), io=12.3MiB (12.9MB), run=120086-120086msec