owned this note   changed 8 months ago  
Published Linked with GitHub

HPC Storage Benchmark for Ling and Grigorios

Conclusions - text suggested for paper

These algorithm benchmarks were collected on the CREATE HPC cluster at King's College London using the scratch storage filesystem. This filesystem uses the Ceph distributed storage system across a large number of storage nodes with diverse disk types and properties.

The CREATE HPC scratch storage in file access benchmarks was found to have a throughput of 645MiB/s sequential read (1179MiB/s for cached files) and a random read latency of 37.8 ms (sdev 51.4 ms) or 12.9 ms (sdev 39.1 ms) for cached files.

Sequential write throughput was 494MiB/s with a random write latency of 48.1 ms (sdev 86.4 ms).

These storage benchmarks were performed during a period of low usage and were found to be subject to occasional degradation due to factors out of our control within the HPC cluster and its Ceph storage cluster.

Notes

Results of tests were surprisingly consistent, though affected by occasional degradations in performance, presumably due to Ceph doing some background task.

How to acknowledge CREATE in papers:
https://docs.er.kcl.ac.uk/CREATE/acknowledging/

Tests

Fio jobfiles mostly as presented at https://docs.oracle.com/en-us/iaas/Content/Block/References/samplefiocommandslinux.htm

Case 1 - sequential read

Jobfile

[global]
bs=256K
iodepth=64
direct=1
ioengine=libaio
group_reporting
time_based
runtime=120
numjobs=4
name=raw-read
rw=read
size=1G
							
[job1]
filename=/scratch/users/k2256813/benchmark/test1

Result 1 - Saturday 8th Feb, lunchtime

job1: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
job1: Laying out IO file (1 file / 1024MiB)
Jobs: 4 (f=4): [R(4)][100.0%][r=1179MiB/s][r=4716 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=4): err= 0: pid=63081: Sat Feb  8 13:38:22 2025
  read: IOPS=4553, BW=1138MiB/s (1194MB/s)(134GiB/120082msec)
    slat (usec): min=51, max=681009, avg=191.42, stdev=1836.70
    clat (usec): min=1722, max=1657.6k, avg=56007.98, stdev=67410.10
     lat (usec): min=1912, max=1657.8k, avg=56199.64, stdev=67428.38
    clat percentiles (msec):
     |  1.00th=[    6],  5.00th=[    8], 10.00th=[    9], 20.00th=[   13],
     | 30.00th=[   18], 40.00th=[   25], 50.00th=[   34], 60.00th=[   45],
     | 70.00th=[   62], 80.00th=[   87], 90.00th=[  129], 95.00th=[  174],
     | 99.00th=[  321], 99.50th=[  388], 99.90th=[  709], 99.95th=[  827],
     | 99.99th=[  927]
   bw (  MiB/s): min=   41, max= 1524, per=100.00%, avg=1138.97, stdev=50.96, samples=960
   iops        : min=  166, max= 6096, avg=4555.70, stdev=203.84, samples=960
  lat (msec)   : 2=0.01%, 4=0.21%, 10=12.49%, 20=21.22%, 50=29.41%
  lat (msec)   : 100=20.72%, 250=13.94%, 500=1.81%, 750=0.11%, 1000=0.07%
  lat (msec)   : 2000=0.01%
  cpu          : usr=0.81%, sys=22.25%, ctx=449921, majf=0, minf=50116
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=546846,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1138MiB/s (1194MB/s), 1138MiB/s-1138MiB/s (1194MB/s-1194MB/s), io=134GiB (143GB), run=120082-120082msec

Result 2 - Sunday 9th Feb, lunchtime

job1: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
Jobs: 4 (f=4): [R(4)][100.0%][r=1178MiB/s][r=4712 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=4): err= 0: pid=234023: Sun Feb  9 12:41:20 2025
  read: IOPS=4333, BW=1083MiB/s (1136MB/s)(127GiB/120201msec)
    slat (usec): min=52, max=245760, avg=226.45, stdev=2547.87
    clat (usec): min=1872, max=5552.0k, avg=58802.42, stdev=127502.23
     lat (msec): min=2, max=5552, avg=59.03, stdev=128.28
    clat percentiles (msec):
     |  1.00th=[    6],  5.00th=[    8], 10.00th=[    9], 20.00th=[   13],
     | 30.00th=[   17], 40.00th=[   23], 50.00th=[   32], 60.00th=[   44],
     | 70.00th=[   61], 80.00th=[   86], 90.00th=[  129], 95.00th=[  178],
     | 99.00th=[  355], 99.50th=[  477], 99.90th=[ 1133], 99.95th=[ 2601],
     | 99.99th=[ 5201]
   bw (  MiB/s): min=    6, max= 1684, per=100.00%, avg=1085.00, stdev=79.24, samples=959
   iops        : min=   24, max= 6736, avg=4339.99, stdev=316.95, samples=959
  lat (msec)   : 2=0.01%, 4=0.16%, 10=13.44%, 20=22.32%, 50=28.50%
  lat (msec)   : 100=19.75%, 250=13.56%, 500=1.83%, 750=0.21%, 1000=0.11%
  lat (msec)   : 2000=0.07%, >=2000=0.06%
  cpu          : usr=0.77%, sys=21.26%, ctx=427902, majf=0, minf=59291
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=520918,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1083MiB/s (1136MB/s), 1083MiB/s-1083MiB/s (1136MB/s-1136MB/s), io=127GiB (137GB), run=120201-120201msec

Case 2 - sequential read, 1M block size

Jobfile

[global]
bs=1M
iodepth=64
direct=1
ioengine=libaio
group_reporting
time_based
runtime=120
numjobs=4
name=raw-read
rw=read
size=1G
							
[job1]
filename=/scratch/users/k2256813/benchmark/test1

Output 1 - Sunday lunchtime

job1: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
Jobs: 4 (f=4): [R(4)][100.0%][r=1175MiB/s][r=1175 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=4): err= 0: pid=234155: Sun Feb  9 12:47:23 2025
  read: IOPS=1147, BW=1148MiB/s (1204MB/s)(135GiB/120232msec)
    slat (usec): min=85, max=5262, avg=227.18, stdev=77.99
    clat (msec): min=4, max=8281, avg=222.71, stdev=594.91
     lat (msec): min=5, max=8282, avg=222.94, stdev=594.91
    clat percentiles (msec):
     |  1.00th=[   11],  5.00th=[   18], 10.00th=[   22], 20.00th=[   32],
     | 30.00th=[   44], 40.00th=[   58], 50.00th=[   83], 60.00th=[  123],
     | 70.00th=[  184], 80.00th=[  271], 90.00th=[  430], 95.00th=[  625],
     | 99.00th=[ 2937], 99.50th=[ 5873], 99.90th=[ 7013], 99.95th=[ 7215],
     | 99.99th=[ 8020]
   bw (  MiB/s): min=  690, max= 1702, per=100.00%, avg=1148.01, stdev=49.84, samples=960
   iops        : min=  690, max= 1702, avg=1148.01, stdev=49.84, samples=960
  lat (msec)   : 10=0.76%, 20=7.06%, 50=27.30%, 100=20.13%, 250=22.73%
  lat (msec)   : 500=14.43%, 750=3.94%, 1000=1.34%, 2000=1.01%, >=2000=1.30%
  cpu          : usr=0.20%, sys=6.89%, ctx=130615, majf=0, minf=98476
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=138013,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1148MiB/s (1204MB/s), 1148MiB/s-1148MiB/s (1204MB/s-1204MB/s), io=135GiB (145GB), run=120232-120232msec

With a new file, to see if it gets put on a different disk

job1: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
job1: Laying out IO file (1 file / 1024MiB)
Jobs: 4 (f=4): [R(4)][100.0%][r=1176MiB/s][r=1176 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=4): err= 0: pid=234249: Sun Feb  9 12:51:00 2025
  read: IOPS=1162, BW=1163MiB/s (1219MB/s)(137GiB/120312msec)
    slat (usec): min=86, max=21862, avg=228.24, stdev=104.83
    clat (msec): min=5, max=10454, avg=219.87, stdev=849.33
     lat (msec): min=5, max=10454, avg=220.10, stdev=849.32
    clat percentiles (msec):
     |  1.00th=[   12],  5.00th=[   16], 10.00th=[   20], 20.00th=[   28],
     | 30.00th=[   37], 40.00th=[   51], 50.00th=[   71], 60.00th=[  105],
     | 70.00th=[  150], 80.00th=[  218], 90.00th=[  342], 95.00th=[  493],
     | 99.00th=[ 3842], 99.50th=[ 8423], 99.90th=[ 9866], 99.95th=[10000],
     | 99.99th=[10402]
   bw (  MiB/s): min=  376, max= 1708, per=100.00%, avg=1163.74, stdev=54.61, samples=960
   iops        : min=  376, max= 1708, avg=1163.49, stdev=54.60, samples=960
  lat (msec)   : 10=0.63%, 20=10.23%, 50=29.09%, 100=18.99%, 250=24.73%
  lat (msec)   : 500=11.50%, 750=2.38%, 1000=0.72%, 2000=0.60%, >=2000=1.13%
  cpu          : usr=0.21%, sys=7.00%, ctx=131824, majf=0, minf=92819
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=139874,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1163MiB/s (1219MB/s), 1163MiB/s-1163MiB/s (1219MB/s-1219MB/s), io=137GiB (147GB), run=120312-120312msec

Case 3 - two nodes at the same time, case 1 and 2 on different files

The results below show that the network connection at the node is the limiting factor for bandwidth

job1: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
Jobs: 1 (f=1): [_(3),R(1)][100.0%][r=182MiB/s][r=728 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=4): err= 0: pid=132630: Sun Feb  9 12:53:45 2025
  read: IOPS=4664, BW=1166MiB/s (1223MB/s)(138GiB/121012msec)
    slat (usec): min=43, max=6016, avg=162.53, stdev=41.76
    clat (usec): min=1702, max=1575.0k, avg=54456.90, stdev=59482.66
     lat (usec): min=1858, max=1575.1k, avg=54619.78, stdev=59481.72
    clat percentiles (msec):
     |  1.00th=[    6],  5.00th=[    8], 10.00th=[   11], 20.00th=[   16],
     | 30.00th=[   23], 40.00th=[   31], 50.00th=[   40], 60.00th=[   50],
     | 70.00th=[   63], 80.00th=[   82], 90.00th=[  113], 95.00th=[  148],
     | 99.00th=[  259], 99.50th=[  326], 99.90th=[  701], 99.95th=[ 1020],
     | 99.99th=[ 1217]
   bw (  MiB/s): min=  810, max= 1468, per=100.00%, avg=1175.75, stdev=27.90, samples=959
   iops        : min= 3242, max= 5874, avg=4702.99, stdev=111.62, samples=959
  lat (msec)   : 2=0.01%, 4=0.19%, 10=8.97%, 20=17.08%, 50=34.36%
  lat (msec)   : 100=26.19%, 250=12.10%, 500=0.95%, 750=0.07%, 1000=0.03%
  lat (msec)   : 2000=0.05%
  cpu          : usr=0.85%, sys=20.14%, ctx=466442, majf=0, minf=54023
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=564452,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1166MiB/s (1223MB/s), 1166MiB/s-1166MiB/s (1223MB/s-1223MB/s), io=138GiB (148GB), run=121012-121012msec

job1: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
Jobs: 4 (f=4): [R(4)][100.0%][r=1182MiB/s][r=1182 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=4): err= 0: pid=234303: Sun Feb  9 12:53:46 2025
  read: IOPS=1177, BW=1177MiB/s (1234MB/s)(138GiB/120282msec)
    slat (usec): min=88, max=5111, avg=226.57, stdev=71.50
    clat (msec): min=6, max=9929, avg=217.17, stdev=805.87
     lat (msec): min=6, max=9929, avg=217.40, stdev=805.87
    clat percentiles (msec):
     |  1.00th=[   14],  5.00th=[   17], 10.00th=[   21], 20.00th=[   29],
     | 30.00th=[   38], 40.00th=[   51], 50.00th=[   71], 60.00th=[  104],
     | 70.00th=[  153], 80.00th=[  220], 90.00th=[  347], 95.00th=[  498],
     | 99.00th=[ 5537], 99.50th=[ 8087], 99.90th=[ 9060], 99.95th=[ 9194],
     | 99.99th=[ 9463]
   bw (  MiB/s): min=  702, max= 1790, per=100.00%, avg=1177.77, stdev=51.14, samples=960
   iops        : min=  702, max= 1790, avg=1177.77, stdev=51.14, samples=960
  lat (msec)   : 10=0.12%, 20=9.43%, 50=30.04%, 100=19.59%, 250=24.10%
  lat (msec)   : 500=11.75%, 750=2.59%, 1000=0.78%, 2000=0.47%, >=2000=1.13%
  cpu          : usr=0.22%, sys=7.02%, ctx=133487, majf=0, minf=82036
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=141584,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1177MiB/s (1234MB/s), 1177MiB/s-1177MiB/s (1234MB/s-1234MB/s), io=138GiB (148GB), run=120282-120282msec

Case 4 - two nodes case 2 on same file

job1: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
Jobs: 4 (f=4): [R(4)][100.0%][r=1175MiB/s][r=1175 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=4): err= 0: pid=234371: Sun Feb  9 12:56:33 2025
  read: IOPS=1177, BW=1178MiB/s (1235MB/s)(138GiB/120238msec)
    slat (usec): min=68, max=4341, avg=227.49, stdev=111.04
    clat (msec): min=6, max=9032, avg=217.09, stdev=744.27
     lat (msec): min=6, max=9032, avg=217.32, stdev=744.27
    clat percentiles (msec):
     |  1.00th=[   13],  5.00th=[   17], 10.00th=[   20], 20.00th=[   27],
     | 30.00th=[   36], 40.00th=[   49], 50.00th=[   68], 60.00th=[  101],
     | 70.00th=[  148], 80.00th=[  218], 90.00th=[  342], 95.00th=[  514],
     | 99.00th=[ 5604], 99.50th=[ 7013], 99.90th=[ 8288], 99.95th=[ 8423],
     | 99.99th=[ 8792]
   bw (  MiB/s): min=  636, max= 1924, per=100.00%, avg=1177.84, stdev=51.23, samples=960
   iops        : min=  636, max= 1924, avg=1177.84, stdev=51.23, samples=960
  lat (msec)   : 10=0.12%, 20=10.82%, 50=30.09%, 100=18.94%, 250=23.61%
  lat (msec)   : 500=11.18%, 750=2.42%, 1000=0.68%, 2000=0.60%, >=2000=1.55%
  cpu          : usr=0.22%, sys=7.05%, ctx=133445, majf=0, minf=196978
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=141593,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1178MiB/s (1235MB/s), 1178MiB/s-1178MiB/s (1235MB/s-1235MB/s), io=138GiB (148GB), run=120238-120238msec

job1: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
Jobs: 4 (f=4): [R(4)][100.0%][r=1183MiB/s][r=1183 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=4): err= 0: pid=132689: Sun Feb  9 12:56:32 2025
  read: IOPS=1117, BW=1118MiB/s (1172MB/s)(131GiB/120339msec)
    slat (usec): min=54, max=195553, avg=421.97, stdev=4088.77
    clat (msec): min=7, max=11231, avg=228.48, stdev=732.36
     lat (msec): min=7, max=11232, avg=228.90, stdev=732.80
    clat percentiles (msec):
     |  1.00th=[   14],  5.00th=[   20], 10.00th=[   26], 20.00th=[   38],
     | 30.00th=[   52], 40.00th=[   71], 50.00th=[   95], 60.00th=[  129],
     | 70.00th=[  174], 80.00th=[  239], 90.00th=[  368], 95.00th=[  523],
     | 99.00th=[ 3641], 99.50th=[ 7215], 99.90th=[ 8356], 99.95th=[ 9060],
     | 99.99th=[10805]
   bw (  MiB/s): min=   52, max= 1770, per=100.00%, avg=1119.05, stdev=72.44, samples=960
   iops        : min=   52, max= 1770, avg=1119.05, stdev=72.44, samples=960
  lat (msec)   : 10=0.07%, 20=5.59%, 50=23.27%, 100=22.69%, 250=29.85%
  lat (msec)   : 500=13.05%, 750=2.73%, 1000=0.81%, 2000=0.61%, >=2000=1.33%
  cpu          : usr=0.30%, sys=6.92%, ctx=125102, majf=0, minf=232459
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=134538,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1118MiB/s (1172MB/s), 1118MiB/s-1118MiB/s (1172MB/s-1172MB/s), io=131GiB (141GB), run=120339-120339msec

Case 5 - seq read 28 processes

Jobfile

[global]
bs=256K
iodepth=64
direct=1
ioengine=libaio
group_reporting
time_based
runtime=120
numjobs=28
name=raw-read
rw=read
size=1G
							
[job1]
filename=/scratch/users/k2256813/benchmark/test1

Output

job1: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 28 processes
Jobs: 28 (f=28): [R(28)][100.0%][r=1179MiB/s][r=4716 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=28): err= 0: pid=234528: Sun Feb  9 13:00:30 2025
  read: IOPS=4706, BW=1177MiB/s (1234MB/s)(138GiB/120485msec)
    slat (usec): min=54, max=5704, avg=181.58, stdev=51.82
    clat (msec): min=2, max=13460, avg=380.00, stdev=1282.10
     lat (msec): min=2, max=13460, avg=380.19, stdev=1282.10
    clat percentiles (msec):
     |  1.00th=[    5],  5.00th=[    7], 10.00th=[    9], 20.00th=[   14],
     | 30.00th=[   21], 40.00th=[   31], 50.00th=[   48], 60.00th=[   80],
     | 70.00th=[  136], 80.00th=[  243], 90.00th=[  558], 95.00th=[ 1720],
     | 99.00th=[ 7752], 99.50th=[10000], 99.90th=[12550], 99.95th=[12818],
     | 99.99th=[13221]
   bw (  MiB/s): min=   20, max= 4566, per=100.00%, avg=1362.19, stdev=33.55, samples=5809
   iops        : min=   80, max=18265, avg=5448.75, stdev=134.19, samples=5809
  lat (msec)   : 4=0.25%, 10=13.40%, 20=16.10%, 50=21.03%, 100=13.63%
  lat (msec)   : 250=16.04%, 500=8.66%, 750=2.80%, 1000=1.35%, 2000=2.08%
  lat (msec)   : >=2000=4.65%
  cpu          : usr=0.13%, sys=3.26%, ctx=544521, majf=0, minf=230030
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.7%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=567111,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1177MiB/s (1234MB/s), 1177MiB/s-1177MiB/s (1234MB/s-1234MB/s), io=138GiB (149GB), run=120485-120485msec

Case 6 - seq read 1 process

This case shows that we can't saturate the bandwidth with 1 fio process

Input

[global]
bs=256K
iodepth=64
direct=1
ioengine=libaio
group_reporting
time_based
runtime=120
numjobs=1
name=raw-read
rw=read
size=1G
							
[job1]
filename=/scratch/users/k2256813/benchmark/test1

Output

job1: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=760MiB/s][r=3039 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=1): err= 0: pid=234628: Sun Feb  9 13:03:17 2025
  read: IOPS=3386, BW=847MiB/s (888MB/s)(99.4GiB/120211msec)
    slat (usec): min=51, max=4843, avg=165.12, stdev=124.09
    clat (usec): min=1823, max=1115.6k, avg=18728.88, stdev=36821.54
     lat (usec): min=1993, max=1115.7k, avg=18894.20, stdev=36813.90
    clat percentiles (msec):
     |  1.00th=[    4],  5.00th=[    5], 10.00th=[    6], 20.00th=[    7],
     | 30.00th=[    8], 40.00th=[    9], 50.00th=[   10], 60.00th=[   11],
     | 70.00th=[   13], 80.00th=[   18], 90.00th=[   36], 95.00th=[   68],
     | 99.00th=[  163], 99.50th=[  222], 99.90th=[  477], 99.95th=[  617],
     | 99.99th=[  827]
   bw (  KiB/s): min=562688, max=1082880, per=100.00%, avg=868390.40, stdev=99229.00, samples=240
   iops        : min= 2198, max= 4230, avg=3392.15, stdev=387.61, samples=240
  lat (msec)   : 2=0.01%, 4=1.98%, 10=52.14%, 20=28.96%, 50=9.76%
  lat (msec)   : 100=4.44%, 250=2.31%, 500=0.31%, 750=0.07%, 1000=0.02%
  lat (msec)   : 2000=0.01%
  cpu          : usr=1.65%, sys=55.86%, ctx=233397, majf=0, minf=17998
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=407121,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=847MiB/s (888MB/s), 847MiB/s-847MiB/s (888MB/s-888MB/s), io=99.4GiB (107GB), run=120211-120211msec

With 2 output - around 13:05

job1: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 2 processes
Jobs: 1 (f=1): [R(1),_(1)][100.0%][eta 00m:00s]                        
job1: (groupid=0, jobs=2): err= 0: pid=132833: Sun Feb  9 13:05:21 2025
  read: IOPS=2780, BW=695MiB/s (729MB/s)(85.6GiB/126138msec)
    slat (usec): min=38, max=7102, avg=160.43, stdev=93.72
    clat (usec): min=655, max=15486k, avg=45376.51, stdev=546472.40
     lat (usec): min=1604, max=15486k, avg=45537.18, stdev=546470.37
    clat percentiles (msec):
     |  1.00th=[    3],  5.00th=[    4], 10.00th=[    5], 20.00th=[    6],
     | 30.00th=[    6], 40.00th=[    7], 50.00th=[    9], 60.00th=[   11],
     | 70.00th=[   15], 80.00th=[   23], 90.00th=[   43], 95.00th=[   74],
     | 99.00th=[  184], 99.50th=[  292], 99.90th=[12953], 99.95th=[13758],
     | 99.99th=[14563]
   bw (  KiB/s): min=26624, max=1438208, per=100.00%, avg=765534.15, stdev=176710.53, samples=469
   iops        : min=  104, max= 5618, avg=2990.36, stdev=690.27, samples=469
  lat (usec)   : 750=0.01%
  lat (msec)   : 2=0.18%, 4=9.27%, 10=48.65%, 20=19.58%, 50=13.97%
  lat (msec)   : 100=5.34%, 250=2.40%, 500=0.19%, 750=0.02%, 1000=0.05%
  lat (msec)   : 2000=0.10%, >=2000=0.25%
  cpu          : usr=0.64%, sys=23.28%, ctx=262203, majf=0, minf=24740
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=350722,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=695MiB/s (729MB/s), 695MiB/s-695MiB/s (729MB/s-729MB/s), io=85.6GiB (91.9GB), run=126138-126138msec

Much slower than expected - why?

With 3 output - around 13:05

job1: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 3 processes
Jobs: 1 (f=1): [_(2),R(1)][100.0%][r=15.8MiB/s][r=63 IOPS][eta 00m:00s]     
job1: (groupid=0, jobs=3): err= 0: pid=234673: Sun Feb  9 13:05:46 2025
  read: IOPS=3375, BW=844MiB/s (885MB/s)(102GiB/124242msec)
    slat (usec): min=46, max=14479, avg=223.24, stdev=139.23
    clat (usec): min=1650, max=15990k, avg=56248.34, stdev=614371.38
     lat (usec): min=1810, max=15990k, avg=56471.82, stdev=614367.78
    clat percentiles (msec):
     |  1.00th=[    3],  5.00th=[    5], 10.00th=[    5], 20.00th=[    7],
     | 30.00th=[    8], 40.00th=[    9], 50.00th=[   11], 60.00th=[   13],
     | 70.00th=[   18], 80.00th=[   27], 90.00th=[   52], 95.00th=[   83],
     | 99.00th=[  188], 99.50th=[  305], 99.90th=[13087], 99.95th=[13758],
     | 99.99th=[14966]
   bw (  KiB/s): min=86528, max=1634304, per=100.00%, avg=901704.97, stdev=104746.15, samples=714
   iops        : min=  338, max= 6384, avg=3522.29, stdev=409.16, samples=714
  lat (msec)   : 2=0.06%, 4=4.43%, 10=45.39%, 20=23.98%, 50=15.91%
  lat (msec)   : 100=6.75%, 250=2.85%, 500=0.21%, 750=0.02%, 1000=0.01%
  lat (msec)   : 2000=0.01%, >=2000=0.39%
  cpu          : usr=0.76%, sys=25.90%, ctx=313501, majf=0, minf=31555
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=419344,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=844MiB/s (885MB/s), 844MiB/s-844MiB/s (885MB/s-885MB/s), io=102GiB (110GB), run=124242-124242msec

Much slower than expected - why?

Repeat of 2

job1: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 2 processes
Jobs: 2 (f=2): [R(2)][100.0%][r=1172MiB/s][r=4688 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=2): err= 0: pid=132885: Sun Feb  9 13:08:13 2025
  read: IOPS=4611, BW=1153MiB/s (1209MB/s)(136GiB/120427msec)
    slat (usec): min=44, max=8028, avg=143.60, stdev=47.43
    clat (usec): min=1587, max=2149.6k, avg=27564.93, stdev=54307.82
     lat (usec): min=1756, max=2149.7k, avg=27708.79, stdev=54306.66
    clat percentiles (msec):
     |  1.00th=[    5],  5.00th=[    6], 10.00th=[    7], 20.00th=[    9],
     | 30.00th=[   11], 40.00th=[   13], 50.00th=[   16], 60.00th=[   20],
     | 70.00th=[   26], 80.00th=[   36], 90.00th=[   56], 95.00th=[   82],
     | 99.00th=[  163], 99.50th=[  226], 99.90th=[  827], 99.95th=[ 1301],
     | 99.99th=[ 1703]
   bw (  MiB/s): min=  538, max= 1485, per=100.00%, avg=1157.16, stdev=62.41, samples=479
   iops        : min= 2152, max= 5940, avg=4628.67, stdev=249.65, samples=479
  lat (msec)   : 2=0.01%, 4=0.94%, 10=27.63%, 20=31.60%, 50=28.04%
  lat (msec)   : 100=8.42%, 250=2.96%, 500=0.25%, 750=0.04%, 1000=0.04%
  lat (msec)   : 2000=0.07%, >=2000=0.01%
  cpu          : usr=1.09%, sys=34.48%, ctx=400565, majf=0, minf=52028
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=555316,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1153MiB/s (1209MB/s), 1153MiB/s-1153MiB/s (1209MB/s-1209MB/s), io=136GiB (146GB), run=120427-120427msec

Repeat of 3

job1: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 3 processes
Jobs: 3 (f=3): [R(3)][100.0%][r=1178MiB/s][r=4710 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=3): err= 0: pid=234725: Sun Feb  9 13:08:05 2025
  read: IOPS=4675, BW=1169MiB/s (1226MB/s)(137GiB/120073msec)
    slat (usec): min=50, max=14698, avg=181.83, stdev=67.89
    clat (usec): min=1839, max=2187.6k, avg=40870.42, stdev=63044.02
     lat (msec): min=2, max=2187, avg=41.05, stdev=63.04
    clat percentiles (msec):
     |  1.00th=[    6],  5.00th=[    7], 10.00th=[    9], 20.00th=[   12],
     | 30.00th=[   16], 40.00th=[   21], 50.00th=[   27], 60.00th=[   34],
     | 70.00th=[   44], 80.00th=[   59], 90.00th=[   86], 95.00th=[  115],
     | 99.00th=[  203], 99.50th=[  262], 99.90th=[  902], 99.95th=[ 1435],
     | 99.99th=[ 2022]
   bw (  MiB/s): min=  694, max= 1596, per=100.00%, avg=1169.26, stdev=44.76, samples=720
   iops        : min= 2776, max= 6386, avg=4677.03, stdev=179.04, samples=720
  lat (msec)   : 2=0.01%, 4=0.17%, 10=14.21%, 20=25.23%, 50=35.24%
  lat (msec)   : 100=18.10%, 250=6.49%, 500=0.38%, 750=0.06%, 1000=0.03%
  lat (msec)   : 2000=0.08%, >=2000=0.01%
  cpu          : usr=1.02%, sys=29.60%, ctx=433163, majf=0, minf=35168
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=561432,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1169MiB/s (1226MB/s), 1169MiB/s-1169MiB/s (1226MB/s-1226MB/s), io=137GiB (147GB), run=120073-120073msec

Repeat of 1 - again doesn't saturate, seems to be genuine

job1: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=813MiB/s][r=3252 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=1): err= 0: pid=234793: Sun Feb  9 13:10:38 2025
  read: IOPS=3327, BW=832MiB/s (872MB/s)(97.6GiB/120131msec)
    slat (usec): min=51, max=4484, avg=167.07, stdev=126.14
    clat (usec): min=1378, max=1915.4k, avg=19060.43, stdev=51280.28
     lat (usec): min=1831, max=1915.5k, avg=19227.67, stdev=51273.92
    clat percentiles (msec):
     |  1.00th=[    4],  5.00th=[    5], 10.00th=[    6], 20.00th=[    7],
     | 30.00th=[    8], 40.00th=[    9], 50.00th=[   10], 60.00th=[   11],
     | 70.00th=[   13], 80.00th=[   17], 90.00th=[   34], 95.00th=[   65],
     | 99.00th=[  163], 99.50th=[  224], 99.90th=[  818], 99.95th=[ 1053],
     | 99.99th=[ 1519]
   bw (  KiB/s): min=466944, max=1054208, per=100.00%, avg=852744.53, stdev=110314.08, samples=240
   iops        : min= 1824, max= 4118, avg=3331.03, stdev=430.91, samples=240
  lat (msec)   : 2=0.02%, 4=2.54%, 10=52.92%, 20=28.42%, 50=9.31%
  lat (msec)   : 100=4.28%, 250=2.10%, 500=0.20%, 750=0.08%, 1000=0.06%
  lat (msec)   : 2000=0.06%
  cpu          : usr=1.67%, sys=55.49%, ctx=226708, majf=0, minf=4111
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=399787,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=832MiB/s (872MB/s), 832MiB/s-832MiB/s (872MB/s-872MB/s), io=97.6GiB (105GB), run=120131-120131msec

Case 7 - random read latency

Jobfile

[global]
bs=4K
iodepth=1
direct=1
ioengine=libaio
group_reporting
time_based
runtime=120
numjobs=1
name=readlatency-test-job
rw=randread
							
[job1]
filename=/scratch/users/k2256813/benchmark/test1

Output

job1: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=284KiB/s][r=71 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=1): err= 0: pid=234847: Sun Feb  9 13:14:02 2025
  read: IOPS=76, BW=306KiB/s (314kB/s)(35.9MiB/120002msec)
    slat (usec): min=127, max=500, avg=169.64, stdev=17.16
    clat (usec): min=731, max=613984, avg=12885.33, stdev=39086.79
     lat (usec): min=881, max=614146, avg=13055.19, stdev=39087.48
    clat percentiles (usec):
     |  1.00th=[  1020],  5.00th=[  1123], 10.00th=[  1188], 20.00th=[  1287],
     | 30.00th=[  1369], 40.00th=[  1532], 50.00th=[  1729], 60.00th=[  1975],
     | 70.00th=[  2311], 80.00th=[  2933], 90.00th=[ 29492], 95.00th=[ 84411],
     | 99.00th=[196084], 99.50th=[248513], 99.90th=[408945], 99.95th=[476054],
     | 99.99th=[616563]
   bw (  KiB/s): min=    8, max=  880, per=100.00%, avg=307.04, stdev=148.48, samples=239
   iops        : min=    2, max=  220, avg=76.76, stdev=37.12, samples=239
  lat (usec)   : 750=0.02%, 1000=0.77%
  lat (msec)   : 2=60.25%, 4=21.36%, 10=2.94%, 20=2.84%, 50=4.23%
  lat (msec)   : 100=3.71%, 250=3.38%, 500=0.47%, 750=0.02%
  cpu          : usr=0.07%, sys=1.36%, ctx=9221, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=9190,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=306KiB/s (314kB/s), 306KiB/s-306KiB/s (314kB/s-314kB/s), io=35.9MiB (37.6MB), run=120002-120002msec

Case 8 - random read bandwidth

Jobfile

[global]
bs=256K
iodepth=64
direct=1
ioengine=libaio
group_reporting
time_based
runtime=120
numjobs=4
name=raw-randread
rw=randread
							
[job1]
filename=/scratch/users/k2256813/benchmark/test1

Output

job1: (g=0): rw=randread, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
Jobs: 2 (f=2): [r(1),_(1),r(1),_(1)][100.0%][r=167MiB/s][r=668 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=4): err= 0: pid=133014: Sun Feb  9 13:18:52 2025
  read: IOPS=4657, BW=1164MiB/s (1221MB/s)(138GiB/120960msec)
    slat (usec): min=59, max=8884, avg=174.61, stdev=33.64
    clat (msec): min=2, max=1956, avg=54.69, stdev=106.58
     lat (msec): min=2, max=1956, avg=54.87, stdev=106.58
    clat percentiles (msec):
     |  1.00th=[    6],  5.00th=[    7], 10.00th=[    8], 20.00th=[   10],
     | 30.00th=[   11], 40.00th=[   14], 50.00th=[   17], 60.00th=[   24],
     | 70.00th=[   37], 80.00th=[   65], 90.00th=[  142], 95.00th=[  239],
     | 99.00th=[  531], 99.50th=[  684], 99.90th=[ 1099], 99.95th=[ 1318],
     | 99.99th=[ 1620]
   bw (  MiB/s): min=  738, max= 1467, per=100.00%, avg=1173.21, stdev=26.20, samples=960
   iops        : min= 2954, max= 5868, avg=4692.84, stdev=104.79, samples=960
  lat (msec)   : 4=0.16%, 10=25.23%, 20=29.85%, 50=20.68%, 100=10.26%
  lat (msec)   : 250=9.30%, 500=3.37%, 750=0.76%, 1000=0.24%, 2000=0.15%
  cpu          : usr=1.02%, sys=21.61%, ctx=438328, majf=0, minf=35451
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=563393,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1164MiB/s (1221MB/s), 1164MiB/s-1164MiB/s (1221MB/s-1221MB/s), io=138GiB (148GB), run=120960-120960msec

Output with 1 process

job1: (g=0): rw=randread, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=1094MiB/s][r=4375 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=1): err= 0: pid=234922: Sun Feb  9 13:20:28 2025
  read: IOPS=4210, BW=1053MiB/s (1104MB/s)(124GiB/120179msec)
    slat (usec): min=51, max=6953, avg=141.15, stdev=42.01
    clat (usec): min=1724, max=1883.1k, avg=15056.63, stdev=41548.21
     lat (usec): min=1864, max=1883.2k, avg=15197.93, stdev=41548.38
    clat percentiles (msec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    4], 20.00th=[    4],
     | 30.00th=[    5], 40.00th=[    5], 50.00th=[    6], 60.00th=[    6],
     | 70.00th=[    7], 80.00th=[   10], 90.00th=[   31], 95.00th=[   69],
     | 99.00th=[  180], 99.50th=[  230], 99.90th=[  456], 99.95th=[  651],
     | 99.99th=[ 1284]
   bw (  MiB/s): min=  764, max= 1157, per=100.00%, avg=1053.96, stdev=63.51, samples=240
   iops        : min= 3056, max= 4628, avg=4215.84, stdev=254.02, samples=240
  lat (msec)   : 2=0.03%, 4=27.40%, 10=53.84%, 20=6.18%, 50=5.57%
  lat (msec)   : 100=4.01%, 250=2.57%, 500=0.32%, 750=0.04%, 1000=0.02%
  lat (msec)   : 2000=0.02%
  cpu          : usr=2.08%, sys=59.57%, ctx=243449, majf=0, minf=19379
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=505964,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1053MiB/s (1104MB/s), 1053MiB/s-1053MiB/s (1104MB/s-1104MB/s), io=124GiB (133GB), run=120179-120179msec

Case 9 - random rw

Input

[global]
bs=256K
iodepth=64
direct=1
ioengine=libaio
group_reporting
time_based
runtime=120
numjobs=4
name=raw-randreadwrite
rw=randrw
size=1G
							
[job1]
filename=/scratch/users/k2256813/benchmark/test3

Output

job1: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
job1: Laying out IO file (1 file / 1024MiB)
Jobs: 4 (f=4): [m(4)][100.0%][eta 00m:00s]                                           
job1: (groupid=0, jobs=4): err= 0: pid=235033: Sun Feb  9 13:26:10 2025
  read: IOPS=1158, BW=290MiB/s (304MB/s)(36.7GiB/129904msec)
    slat (usec): min=50, max=1969, avg=198.47, stdev=48.49
    clat (usec): min=1717, max=15051k, avg=92206.70, stdev=458379.24
     lat (usec): min=1899, max=15052k, avg=92405.50, stdev=458380.92
    clat percentiles (msec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    4],
     | 30.00th=[    5], 40.00th=[    6], 50.00th=[   10], 60.00th=[   20],
     | 70.00th=[   36], 80.00th=[   71], 90.00th=[  174], 95.00th=[  334],
     | 99.00th=[ 1116], 99.50th=[ 2400], 99.90th=[ 6812], 99.95th=[ 9597],
     | 99.99th=[12818]
   bw (  KiB/s): min=159744, max=479744, per=100.00%, avg=320801.74, stdev=14656.32, samples=960
   iops        : min=  624, max= 1874, avg=1253.11, stdev=57.25, samples=960
  write: IOPS=1159, BW=290MiB/s (304MB/s)(36.8GiB/129904msec); 0 zone resets
    slat (usec): min=58, max=4829, avg=239.87, stdev=109.04
    clat (msec): min=5, max=15012, avg=127.26, stdev=499.30
     lat (msec): min=5, max=15013, avg=127.50, stdev=499.30
    clat percentiles (msec):
     |  1.00th=[   10],  5.00th=[   15], 10.00th=[   18], 20.00th=[   22],
     | 30.00th=[   27], 40.00th=[   32], 50.00th=[   39], 60.00th=[   51],
     | 70.00th=[   72], 80.00th=[  114], 90.00th=[  226], 95.00th=[  401],
     | 99.00th=[ 1217], 99.50th=[ 2802], 99.90th=[ 9194], 99.95th=[ 9731],
     | 99.99th=[12818]
   bw (  KiB/s): min=169472, max=490496, per=100.00%, avg=321100.32, stdev=14166.73, samples=960
   iops        : min=  662, max= 1916, avg=1254.28, stdev=55.34, samples=960
  lat (msec)   : 2=0.07%, 4=14.69%, 10=10.89%, 20=12.91%, 50=28.83%
  lat (msec)   : 100=13.59%, 250=11.08%, 500=4.56%, 750=1.47%, 1000=0.66%
  lat (msec)   : 2000=0.66%, >=2000=0.57%
  cpu          : usr=1.23%, sys=12.33%, ctx=257127, majf=0, minf=16669
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=150473,150667,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=290MiB/s (304MB/s), 290MiB/s-290MiB/s (304MB/s-304MB/s), io=36.7GiB (39.4GB), run=129904-129904msec
  WRITE: bw=290MiB/s (304MB/s), 290MiB/s-290MiB/s (304MB/s-304MB/s), io=36.8GiB (39.5GB), run=129904-129904msec

Does this show that performance of other benchmarks shows caching?
Test with a much larger file that won't be cached

Case 10 - 100GB file seq read

Jobfile

[global]
bs=256K
iodepth=64
direct=1
ioengine=libaio
group_reporting
time_based
runtime=60
numjobs=4
name=raw-read
rw=read
size=100G
							
[job1]
filename=/scratch/users/k2256813/benchmark/test_100G

Output

job1: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
job1: Laying out IO file (1 file / 102400MiB)
Jobs: 4 (f=4): [R(4)][100.0%][r=585MiB/s][r=2341 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=4): err= 0: pid=235789: Sun Feb  9 13:33:08 2025
  read: IOPS=2580, BW=645MiB/s (677MB/s)(38.0GiB/60326msec)
    slat (usec): min=54, max=8994, avg=227.18, stdev=188.56
    clat (usec): min=1661, max=1025.0k, avg=98924.95, stdev=136291.56
     lat (usec): min=1833, max=1025.2k, avg=99152.32, stdev=136261.29
    clat percentiles (msec):
     |  1.00th=[    4],  5.00th=[    5], 10.00th=[    7], 20.00th=[   12],
     | 30.00th=[   19], 40.00th=[   28], 50.00th=[   42], 60.00th=[   59],
     | 70.00th=[   85], 80.00th=[  171], 90.00th=[  296], 95.00th=[  401],
     | 99.00th=[  600], 99.50th=[  693], 99.90th=[  877], 99.95th=[  919],
     | 99.99th=[ 1011]
   bw (  KiB/s): min=250880, max=1191936, per=100.00%, avg=663210.67, stdev=41274.75, samples=480
   iops        : min=  980, max= 4656, avg=2590.68, stdev=161.23, samples=480
  lat (msec)   : 2=0.07%, 4=2.65%, 10=15.38%, 20=13.39%, 50=23.75%
  lat (msec)   : 100=17.65%, 250=14.05%, 500=10.69%, 750=2.09%, 1000=0.27%
  lat (msec)   : 2000=0.02%
  cpu          : usr=0.44%, sys=14.92%, ctx=124150, majf=0, minf=27026
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=155695,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=645MiB/s (677MB/s), 645MiB/s-645MiB/s (677MB/s-677MB/s), io=38.0GiB (40.8GB), run=60326-60326msec

Repeat after deleting file

job1: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
job1: Laying out IO file (1 file / 102400MiB)
Jobs: 4 (f=4): [R(4)][9.3%][r=771MiB/s][r=3085 IOPS][eta 09m:56s]  
job1: (groupid=0, jobs=4): err= 0: pid=235900: Sun Feb  9 13:36:44 2025
  read: IOPS=2531, BW=633MiB/s (664MB/s)(37.2GiB/60201msec)
    slat (usec): min=53, max=7276, avg=224.25, stdev=174.54
    clat (usec): min=1647, max=1528.2k, avg=100870.34, stdev=136508.59
     lat (usec): min=1831, max=1528.3k, avg=101094.83, stdev=136480.13
    clat percentiles (msec):
     |  1.00th=[    4],  5.00th=[    6], 10.00th=[    7], 20.00th=[   12],
     | 30.00th=[   21], 40.00th=[   32], 50.00th=[   48], 60.00th=[   69],
     | 70.00th=[  101], 80.00th=[  167], 90.00th=[  275], 95.00th=[  393],
     | 99.00th=[  609], 99.50th=[  718], 99.90th=[  978], 99.95th=[ 1385],
     | 99.99th=[ 1519]
   bw (  KiB/s): min=296960, max=1181184, per=100.00%, avg=649102.70, stdev=43387.20, samples=480
   iops        : min= 1160, max= 4614, avg=2535.53, stdev=169.47, samples=480
  lat (msec)   : 2=0.03%, 4=2.19%, 10=14.94%, 20=12.22%, 50=21.83%
  lat (msec)   : 100=18.77%, 250=18.27%, 500=9.51%, 750=1.78%, 1000=0.38%
  lat (msec)   : 2000=0.09%
  cpu          : usr=0.46%, sys=14.52%, ctx=123222, majf=0, minf=29709
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=152389,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=633MiB/s (664MB/s), 633MiB/s-633MiB/s (664MB/s-664MB/s), io=37.2GiB (39.9GB), run=60201-60201msec

Repeat keeping file

job1: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
Jobs: 4 (f=4): [R(4)][100.0%][r=658MiB/s][r=2632 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=4): err= 0: pid=235934: Sun Feb  9 13:37:53 2025
  read: IOPS=3818, BW=955MiB/s (1001MB/s)(56.2GiB/60253msec)
    slat (usec): min=52, max=14983, avg=199.21, stdev=113.11
    clat (usec): min=1652, max=1425.2k, avg=66776.98, stdev=95989.52
     lat (usec): min=1815, max=1425.4k, avg=66976.43, stdev=95978.41
    clat percentiles (msec):
     |  1.00th=[    4],  5.00th=[    7], 10.00th=[    8], 20.00th=[   12],
     | 30.00th=[   17], 40.00th=[   24], 50.00th=[   33], 60.00th=[   45],
     | 70.00th=[   64], 80.00th=[   95], 90.00th=[  161], 95.00th=[  255],
     | 99.00th=[  477], 99.50th=[  567], 99.90th=[  810], 99.95th=[  919],
     | 99.99th=[  995]
   bw (  KiB/s): min=313344, max=1880064, per=100.00%, avg=980548.27, stdev=78527.08, samples=480
   iops        : min= 1224, max= 7344, avg=3830.27, stdev=306.75, samples=480
  lat (msec)   : 2=0.01%, 4=1.11%, 10=14.62%, 20=19.72%, 50=28.10%
  lat (msec)   : 100=17.62%, 250=13.63%, 500=4.39%, 750=0.64%, 1000=0.15%
  lat (msec)   : 2000=0.01%
  cpu          : usr=0.69%, sys=19.77%, ctx=186279, majf=0, minf=51174
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=230068,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=955MiB/s (1001MB/s), 955MiB/s-955MiB/s (1001MB/s-1001MB/s), io=56.2GiB (60.3GB), run=60253-60253msec

Repeat after deleting file

job1: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
job1: Laying out IO file (1 file / 102400MiB)
Jobs: 4 (f=4): [R(4)][100.0%][r=707MiB/s][r=2829 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=4): err= 0: pid=236216: Sun Feb  9 13:47:03 2025
  read: IOPS=2707, BW=677MiB/s (710MB/s)(39.9GiB/60389msec)
    slat (usec): min=48, max=7927, avg=225.54, stdev=174.79
    clat (usec): min=1648, max=1234.9k, avg=94278.95, stdev=133825.06
     lat (usec): min=1807, max=1235.1k, avg=94504.71, stdev=133797.42
    clat percentiles (msec):
     |  1.00th=[    4],  5.00th=[    6], 10.00th=[    7], 20.00th=[   12],
     | 30.00th=[   20], 40.00th=[   29], 50.00th=[   42], 60.00th=[   57],
     | 70.00th=[   81], 80.00th=[  150], 90.00th=[  275], 95.00th=[  372],
     | 99.00th=[  634], 99.50th=[  743], 99.90th=[  969], 99.95th=[ 1028],
     | 99.99th=[ 1200]
   bw (  KiB/s): min=276791, max=1096192, per=100.00%, avg=696569.50, stdev=41634.45, samples=480
   iops        : min= 1081, max= 4282, avg=2720.97, stdev=162.64, samples=480
  lat (msec)   : 2=0.05%, 4=2.30%, 10=15.25%, 20=13.27%, 50=25.11%
  lat (msec)   : 100=18.11%, 250=14.54%, 500=9.09%, 750=1.82%, 1000=0.39%
  lat (msec)   : 2000=0.06%
  cpu          : usr=0.44%, sys=15.65%, ctx=131238, majf=0, minf=24737
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=163513,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=677MiB/s (710MB/s), 677MiB/s-677MiB/s (710MB/s-710MB/s), io=39.9GiB (42.9GB), run=60389-60389msec

Case 11 - 100GB random read

Jobfile

[global]
bs=256K
iodepth=64
direct=1
ioengine=libaio
group_reporting
time_based
runtime=60
numjobs=4
name=raw-read
rw=randread
size=100G
							
[job1]
filename=/scratch/users/k2256813/benchmark/test_100G

Output

job1: (g=0): rw=randread, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
job1: Laying out IO file (1 file / 102400MiB)
Jobs: 4 (f=4): [r(4)][16.6%][r=1159MiB/s][r=4637 IOPS][eta 05m:06s] 
job1: (groupid=0, jobs=4): err= 0: pid=236025: Sun Feb  9 13:42:50 2025
  read: IOPS=4531, BW=1133MiB/s (1188MB/s)(66.8GiB/60391msec)
    slat (usec): min=53, max=1090, avg=189.44, stdev=27.98
    clat (msec): min=2, max=2202, avg=56.20, stdev=83.07
     lat (msec): min=2, max=2203, avg=56.39, stdev=83.07
    clat percentiles (msec):
     |  1.00th=[    6],  5.00th=[    7], 10.00th=[    9], 20.00th=[   15],
     | 30.00th=[   20], 40.00th=[   24], 50.00th=[   29], 60.00th=[   36],
     | 70.00th=[   50], 80.00th=[   73], 90.00th=[  130], 95.00th=[  220],
     | 99.00th=[  397], 99.50th=[  514], 99.90th=[  793], 99.95th=[  969],
     | 99.99th=[ 1536]
   bw (  MiB/s): min=  840, max= 1305, per=100.00%, avg=1139.16, stdev=25.40, samples=480
   iops        : min= 3360, max= 5222, avg=4556.63, stdev=101.60, samples=480
  lat (msec)   : 4=0.14%, 10=12.32%, 20=19.50%, 50=38.26%, 100=16.12%
  lat (msec)   : 250=10.28%, 500=2.84%, 750=0.42%, 1000=0.07%, 2000=0.05%
  lat (msec)   : >=2000=0.01%
  cpu          : usr=0.92%, sys=22.46%, ctx=218438, majf=0, minf=16697
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=273650,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1133MiB/s (1188MB/s), 1133MiB/s-1133MiB/s (1188MB/s-1188MB/s), io=66.8GiB (71.7GB), run=60391-60391msec

This is faster than I expected - were the previous tests slowed down?

Case 12 - random rw latency

Input

Output

job1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [m(1)][100.0%][r=24KiB/s,w=24KiB/s][r=6,w=6 IOPS][eta 00m:00s]   
job1: (groupid=0, jobs=1): err= 0: pid=236267: Sun Feb  9 13:51:17 2025
  read: IOPS=14, BW=58.7KiB/s (60.1kB/s)(7048KiB/120075msec)
    slat (usec): min=130, max=756, avg=173.90, stdev=26.19
    clat (usec): min=891, max=417542, avg=17532.07, stdev=42407.83
     lat (usec): min=1055, max=417716, avg=17706.18, stdev=42409.15
    clat percentiles (usec):
     |  1.00th=[  1045],  5.00th=[  1156], 10.00th=[  1221], 20.00th=[  1319],
     | 30.00th=[  1467], 40.00th=[  1729], 50.00th=[  2089], 60.00th=[  2638],
     | 70.00th=[  8717], 80.00th=[ 17695], 90.00th=[ 47449], 95.00th=[ 89654],
     | 99.00th=[233833], 99.50th=[287310], 99.90th=[367002], 99.95th=[417334],
     | 99.99th=[417334]
   bw (  KiB/s): min=    8, max=  200, per=100.00%, avg=61.29, stdev=38.82, samples=230
   iops        : min=    2, max=   50, avg=15.32, stdev= 9.71, samples=230
  write: IOPS=15, BW=61.3KiB/s (62.7kB/s)(7356KiB/120075msec); 0 zone resets
    slat (usec): min=152, max=109592, avg=236.55, stdev=2551.55
    clat (msec): min=2, max=722, avg=48.09, stdev=86.36
     lat (msec): min=2, max=722, avg=48.32, stdev=86.37
    clat percentiles (msec):
     |  1.00th=[    3],  5.00th=[    4], 10.00th=[    4], 20.00th=[    5],
     | 30.00th=[    6], 40.00th=[    6], 50.00th=[    8], 60.00th=[   17],
     | 70.00th=[   29], 80.00th=[   69], 90.00th=[  163], 95.00th=[  247],
     | 99.00th=[  393], 99.50th=[  460], 99.90th=[  592], 99.95th=[  726],
     | 99.99th=[  726]
   bw (  KiB/s): min=    8, max=  176, per=99.57%, avg=61.52, stdev=34.02, samples=239
   iops        : min=    2, max=   44, avg=15.38, stdev= 8.50, samples=239
  lat (usec)   : 1000=0.14%
  lat (msec)   : 2=23.30%, 4=16.36%, 10=22.02%, 20=10.91%, 50=10.64%
  lat (msec)   : 100=6.55%, 250=7.25%, 500=2.64%, 750=0.19%
  cpu          : usr=0.02%, sys=0.56%, ctx=3603, majf=0, minf=15
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1762,1839,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=58.7KiB/s (60.1kB/s), 58.7KiB/s-58.7KiB/s (60.1kB/s-60.1kB/s), io=7048KiB (7217kB), run=120075-120075msec
  WRITE: bw=61.3KiB/s (62.7kB/s), 61.3KiB/s-61.3KiB/s (62.7kB/s-62.7kB/s), io=7356KiB (7533kB), run=120075-120075msec

Case 13 -

Input

[global]
bs=256K
iodepth=64
direct=1
ioengine=libaio
group_reporting
time_based
runtime=120
numjobs=4
name=raw-write
rw=write
size=1G
							
[job1]
filename=/scratch/users/k2256813/benchmark/test6

Output

job1: (g=0): rw=write, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
job1: Laying out IO file (1 file / 1024MiB)
job1: Laying out IO file (1 file / 1024MiB)
job1: Laying out IO file (1 file / 1024MiB)
job1: Laying out IO file (1 file / 1024MiB)
Jobs: 4 (f=4): [W(4)][100.0%][w=507MiB/s][w=2028 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=4): err= 0: pid=134086: Sun Feb  9 13:53:25 2025
  write: IOPS=1976, BW=494MiB/s (518MB/s)(58.0GiB/120240msec); 0 zone resets
    slat (usec): min=50, max=14511, avg=228.18, stdev=388.04
    clat (msec): min=3, max=1838, avg=129.21, stdev=112.81
     lat (msec): min=3, max=1838, avg=129.44, stdev=112.81
    clat percentiles (msec):
     |  1.00th=[   20],  5.00th=[   33], 10.00th=[   41], 20.00th=[   55],
     | 30.00th=[   69], 40.00th=[   83], 50.00th=[   99], 60.00th=[  116],
     | 70.00th=[  140], 80.00th=[  178], 90.00th=[  253], 95.00th=[  334],
     | 99.00th=[  575], 99.50th=[  693], 99.90th=[ 1062], 99.95th=[ 1217],
     | 99.99th=[ 1519]
   bw (  KiB/s): min=210432, max=797184, per=100.00%, avg=506612.46, stdev=25013.38, samples=960
   iops        : min=  822, max= 3114, avg=1978.83, stdev=97.73, samples=960
  lat (msec)   : 4=0.01%, 10=0.14%, 20=1.01%, 50=15.55%, 100=34.54%
  lat (msec)   : 250=38.47%, 500=8.77%, 750=1.15%, 1000=0.25%, 2000=0.12%
  cpu          : usr=1.47%, sys=9.46%, ctx=64617, majf=0, minf=39458
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,237694,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=494MiB/s (518MB/s), 494MiB/s-494MiB/s (518MB/s-518MB/s), io=58.0GiB (62.3GB), run=120240-120240msec

Case 14 - random read latency on 100G file

Input

[global]
bs=4K
iodepth=1
direct=1
ioengine=libaio
group_reporting
time_based
runtime=120
numjobs=1
name=readlatency-test-job
rw=randread
size=100G
							
[job1]
filename=/scratch/users/k2256813/benchmark/test_100G

Output

job1: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.28
Starting 1 process
job1: Laying out IO file (1 file / 102400MiB)
Jobs: 1 (f=1): [r(1)][100.0%][r=148KiB/s][r=37 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=1): err= 0: pid=236406: Sun Feb  9 13:59:13 2025
  read: IOPS=26, BW=105KiB/s (108kB/s)(12.3MiB/120086msec)
    slat (usec): min=107, max=325, avg=178.93, stdev=27.32
    clat (usec): min=1709, max=869548, avg=37807.73, stdev=51401.37
     lat (usec): min=1871, max=869718, avg=37986.95, stdev=51402.18
    clat percentiles (msec):
     |  1.00th=[    7],  5.00th=[   12], 10.00th=[   14], 20.00th=[   16],
     | 30.00th=[   18], 40.00th=[   20], 50.00th=[   21], 60.00th=[   23],
     | 70.00th=[   29], 80.00th=[   43], 90.00th=[   81], 95.00th=[  128],
     | 99.00th=[  257], 99.50th=[  338], 99.90th=[  468], 99.95th=[  709],
     | 99.99th=[  869]
   bw (  KiB/s): min=    8, max=  200, per=100.00%, avg=106.22, stdev=35.73, samples=238
   iops        : min=    2, max=   50, avg=26.55, stdev= 8.93, samples=238
  lat (msec)   : 2=0.13%, 4=0.54%, 10=2.21%, 20=43.09%, 50=37.17%
  lat (msec)   : 100=9.36%, 250=6.42%, 500=0.98%, 750=0.06%, 1000=0.03%
  cpu          : usr=0.02%, sys=0.50%, ctx=3161, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3161,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=105KiB/s (108kB/s), 105KiB/s-105KiB/s (108kB/s-108kB/s), io=12.3MiB (12.9MB), run=120086-120086msec
Select a repo