# HPC Storage Benchmark for Ling and Grigorios
## Conclusions - text suggested for paper
> These algorithm benchmarks were collected on the CREATE HPC cluster at King's College London using the scratch storage filesystem. This filesystem uses the Ceph distributed storage system across a large number of storage nodes with diverse disk types and properties.
>
> The CREATE HPC scratch storage in file access benchmarks was found to have a throughput of 645MiB/s sequential read (1179MiB/s for cached files) and a random read latency of 37.8 ms (sdev 51.4 ms) or 12.9 ms (sdev 39.1 ms) for cached files.
>
> Sequential write throughput was 494MiB/s with a random write latency of 48.1 ms (sdev 86.4 ms).
>
> These storage benchmarks were performed during a period of low usage and were found to be subject to occasional degradation due to factors out of our control within the HPC cluster and its Ceph storage cluster.
## Notes
Results of tests were surprisingly consistent, though affected by occasional degradations in performance, presumably due to Ceph doing some background task.
How to acknowledge CREATE in papers:
https://docs.er.kcl.ac.uk/CREATE/acknowledging/
## Tests
Fio jobfiles mostly as presented at https://docs.oracle.com/en-us/iaas/Content/Block/References/samplefiocommandslinux.htm
### Case 1 - sequential read
Jobfile
```
[global]
bs=256K
iodepth=64
direct=1
ioengine=libaio
group_reporting
time_based
runtime=120
numjobs=4
name=raw-read
rw=read
size=1G
[job1]
filename=/scratch/users/k2256813/benchmark/test1
```
Result 1 - Saturday 8th Feb, lunchtime
```
job1: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
job1: Laying out IO file (1 file / 1024MiB)
Jobs: 4 (f=4): [R(4)][100.0%][r=1179MiB/s][r=4716 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=4): err= 0: pid=63081: Sat Feb 8 13:38:22 2025
read: IOPS=4553, BW=1138MiB/s (1194MB/s)(134GiB/120082msec)
slat (usec): min=51, max=681009, avg=191.42, stdev=1836.70
clat (usec): min=1722, max=1657.6k, avg=56007.98, stdev=67410.10
lat (usec): min=1912, max=1657.8k, avg=56199.64, stdev=67428.38
clat percentiles (msec):
| 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 13],
| 30.00th=[ 18], 40.00th=[ 25], 50.00th=[ 34], 60.00th=[ 45],
| 70.00th=[ 62], 80.00th=[ 87], 90.00th=[ 129], 95.00th=[ 174],
| 99.00th=[ 321], 99.50th=[ 388], 99.90th=[ 709], 99.95th=[ 827],
| 99.99th=[ 927]
bw ( MiB/s): min= 41, max= 1524, per=100.00%, avg=1138.97, stdev=50.96, samples=960
iops : min= 166, max= 6096, avg=4555.70, stdev=203.84, samples=960
lat (msec) : 2=0.01%, 4=0.21%, 10=12.49%, 20=21.22%, 50=29.41%
lat (msec) : 100=20.72%, 250=13.94%, 500=1.81%, 750=0.11%, 1000=0.07%
lat (msec) : 2000=0.01%
cpu : usr=0.81%, sys=22.25%, ctx=449921, majf=0, minf=50116
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=546846,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=1138MiB/s (1194MB/s), 1138MiB/s-1138MiB/s (1194MB/s-1194MB/s), io=134GiB (143GB), run=120082-120082msec
```
Result 2 - Sunday 9th Feb, lunchtime
```
job1: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
Jobs: 4 (f=4): [R(4)][100.0%][r=1178MiB/s][r=4712 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=4): err= 0: pid=234023: Sun Feb 9 12:41:20 2025
read: IOPS=4333, BW=1083MiB/s (1136MB/s)(127GiB/120201msec)
slat (usec): min=52, max=245760, avg=226.45, stdev=2547.87
clat (usec): min=1872, max=5552.0k, avg=58802.42, stdev=127502.23
lat (msec): min=2, max=5552, avg=59.03, stdev=128.28
clat percentiles (msec):
| 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 13],
| 30.00th=[ 17], 40.00th=[ 23], 50.00th=[ 32], 60.00th=[ 44],
| 70.00th=[ 61], 80.00th=[ 86], 90.00th=[ 129], 95.00th=[ 178],
| 99.00th=[ 355], 99.50th=[ 477], 99.90th=[ 1133], 99.95th=[ 2601],
| 99.99th=[ 5201]
bw ( MiB/s): min= 6, max= 1684, per=100.00%, avg=1085.00, stdev=79.24, samples=959
iops : min= 24, max= 6736, avg=4339.99, stdev=316.95, samples=959
lat (msec) : 2=0.01%, 4=0.16%, 10=13.44%, 20=22.32%, 50=28.50%
lat (msec) : 100=19.75%, 250=13.56%, 500=1.83%, 750=0.21%, 1000=0.11%
lat (msec) : 2000=0.07%, >=2000=0.06%
cpu : usr=0.77%, sys=21.26%, ctx=427902, majf=0, minf=59291
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=520918,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=1083MiB/s (1136MB/s), 1083MiB/s-1083MiB/s (1136MB/s-1136MB/s), io=127GiB (137GB), run=120201-120201msec
```
### Case 2 - sequential read, 1M block size
Jobfile
```
[global]
bs=1M
iodepth=64
direct=1
ioengine=libaio
group_reporting
time_based
runtime=120
numjobs=4
name=raw-read
rw=read
size=1G
[job1]
filename=/scratch/users/k2256813/benchmark/test1
```
Output 1 - Sunday lunchtime
```
job1: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
Jobs: 4 (f=4): [R(4)][100.0%][r=1175MiB/s][r=1175 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=4): err= 0: pid=234155: Sun Feb 9 12:47:23 2025
read: IOPS=1147, BW=1148MiB/s (1204MB/s)(135GiB/120232msec)
slat (usec): min=85, max=5262, avg=227.18, stdev=77.99
clat (msec): min=4, max=8281, avg=222.71, stdev=594.91
lat (msec): min=5, max=8282, avg=222.94, stdev=594.91
clat percentiles (msec):
| 1.00th=[ 11], 5.00th=[ 18], 10.00th=[ 22], 20.00th=[ 32],
| 30.00th=[ 44], 40.00th=[ 58], 50.00th=[ 83], 60.00th=[ 123],
| 70.00th=[ 184], 80.00th=[ 271], 90.00th=[ 430], 95.00th=[ 625],
| 99.00th=[ 2937], 99.50th=[ 5873], 99.90th=[ 7013], 99.95th=[ 7215],
| 99.99th=[ 8020]
bw ( MiB/s): min= 690, max= 1702, per=100.00%, avg=1148.01, stdev=49.84, samples=960
iops : min= 690, max= 1702, avg=1148.01, stdev=49.84, samples=960
lat (msec) : 10=0.76%, 20=7.06%, 50=27.30%, 100=20.13%, 250=22.73%
lat (msec) : 500=14.43%, 750=3.94%, 1000=1.34%, 2000=1.01%, >=2000=1.30%
cpu : usr=0.20%, sys=6.89%, ctx=130615, majf=0, minf=98476
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=138013,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=1148MiB/s (1204MB/s), 1148MiB/s-1148MiB/s (1204MB/s-1204MB/s), io=135GiB (145GB), run=120232-120232msec
```
With a new file, to see if it gets put on a different disk
```
job1: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
job1: Laying out IO file (1 file / 1024MiB)
Jobs: 4 (f=4): [R(4)][100.0%][r=1176MiB/s][r=1176 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=4): err= 0: pid=234249: Sun Feb 9 12:51:00 2025
read: IOPS=1162, BW=1163MiB/s (1219MB/s)(137GiB/120312msec)
slat (usec): min=86, max=21862, avg=228.24, stdev=104.83
clat (msec): min=5, max=10454, avg=219.87, stdev=849.33
lat (msec): min=5, max=10454, avg=220.10, stdev=849.32
clat percentiles (msec):
| 1.00th=[ 12], 5.00th=[ 16], 10.00th=[ 20], 20.00th=[ 28],
| 30.00th=[ 37], 40.00th=[ 51], 50.00th=[ 71], 60.00th=[ 105],
| 70.00th=[ 150], 80.00th=[ 218], 90.00th=[ 342], 95.00th=[ 493],
| 99.00th=[ 3842], 99.50th=[ 8423], 99.90th=[ 9866], 99.95th=[10000],
| 99.99th=[10402]
bw ( MiB/s): min= 376, max= 1708, per=100.00%, avg=1163.74, stdev=54.61, samples=960
iops : min= 376, max= 1708, avg=1163.49, stdev=54.60, samples=960
lat (msec) : 10=0.63%, 20=10.23%, 50=29.09%, 100=18.99%, 250=24.73%
lat (msec) : 500=11.50%, 750=2.38%, 1000=0.72%, 2000=0.60%, >=2000=1.13%
cpu : usr=0.21%, sys=7.00%, ctx=131824, majf=0, minf=92819
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=139874,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=1163MiB/s (1219MB/s), 1163MiB/s-1163MiB/s (1219MB/s-1219MB/s), io=137GiB (147GB), run=120312-120312msec
```
### Case 3 - two nodes at the same time, case 1 and 2 on different files
The results below show that the network connection at the node is the limiting factor for bandwidth
```
job1: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
Jobs: 1 (f=1): [_(3),R(1)][100.0%][r=182MiB/s][r=728 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=4): err= 0: pid=132630: Sun Feb 9 12:53:45 2025
read: IOPS=4664, BW=1166MiB/s (1223MB/s)(138GiB/121012msec)
slat (usec): min=43, max=6016, avg=162.53, stdev=41.76
clat (usec): min=1702, max=1575.0k, avg=54456.90, stdev=59482.66
lat (usec): min=1858, max=1575.1k, avg=54619.78, stdev=59481.72
clat percentiles (msec):
| 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 11], 20.00th=[ 16],
| 30.00th=[ 23], 40.00th=[ 31], 50.00th=[ 40], 60.00th=[ 50],
| 70.00th=[ 63], 80.00th=[ 82], 90.00th=[ 113], 95.00th=[ 148],
| 99.00th=[ 259], 99.50th=[ 326], 99.90th=[ 701], 99.95th=[ 1020],
| 99.99th=[ 1217]
bw ( MiB/s): min= 810, max= 1468, per=100.00%, avg=1175.75, stdev=27.90, samples=959
iops : min= 3242, max= 5874, avg=4702.99, stdev=111.62, samples=959
lat (msec) : 2=0.01%, 4=0.19%, 10=8.97%, 20=17.08%, 50=34.36%
lat (msec) : 100=26.19%, 250=12.10%, 500=0.95%, 750=0.07%, 1000=0.03%
lat (msec) : 2000=0.05%
cpu : usr=0.85%, sys=20.14%, ctx=466442, majf=0, minf=54023
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=564452,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=1166MiB/s (1223MB/s), 1166MiB/s-1166MiB/s (1223MB/s-1223MB/s), io=138GiB (148GB), run=121012-121012msec
```
```
job1: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
Jobs: 4 (f=4): [R(4)][100.0%][r=1182MiB/s][r=1182 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=4): err= 0: pid=234303: Sun Feb 9 12:53:46 2025
read: IOPS=1177, BW=1177MiB/s (1234MB/s)(138GiB/120282msec)
slat (usec): min=88, max=5111, avg=226.57, stdev=71.50
clat (msec): min=6, max=9929, avg=217.17, stdev=805.87
lat (msec): min=6, max=9929, avg=217.40, stdev=805.87
clat percentiles (msec):
| 1.00th=[ 14], 5.00th=[ 17], 10.00th=[ 21], 20.00th=[ 29],
| 30.00th=[ 38], 40.00th=[ 51], 50.00th=[ 71], 60.00th=[ 104],
| 70.00th=[ 153], 80.00th=[ 220], 90.00th=[ 347], 95.00th=[ 498],
| 99.00th=[ 5537], 99.50th=[ 8087], 99.90th=[ 9060], 99.95th=[ 9194],
| 99.99th=[ 9463]
bw ( MiB/s): min= 702, max= 1790, per=100.00%, avg=1177.77, stdev=51.14, samples=960
iops : min= 702, max= 1790, avg=1177.77, stdev=51.14, samples=960
lat (msec) : 10=0.12%, 20=9.43%, 50=30.04%, 100=19.59%, 250=24.10%
lat (msec) : 500=11.75%, 750=2.59%, 1000=0.78%, 2000=0.47%, >=2000=1.13%
cpu : usr=0.22%, sys=7.02%, ctx=133487, majf=0, minf=82036
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=141584,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=1177MiB/s (1234MB/s), 1177MiB/s-1177MiB/s (1234MB/s-1234MB/s), io=138GiB (148GB), run=120282-120282msec
```
### Case 4 - two nodes case 2 on same file
```
job1: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
Jobs: 4 (f=4): [R(4)][100.0%][r=1175MiB/s][r=1175 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=4): err= 0: pid=234371: Sun Feb 9 12:56:33 2025
read: IOPS=1177, BW=1178MiB/s (1235MB/s)(138GiB/120238msec)
slat (usec): min=68, max=4341, avg=227.49, stdev=111.04
clat (msec): min=6, max=9032, avg=217.09, stdev=744.27
lat (msec): min=6, max=9032, avg=217.32, stdev=744.27
clat percentiles (msec):
| 1.00th=[ 13], 5.00th=[ 17], 10.00th=[ 20], 20.00th=[ 27],
| 30.00th=[ 36], 40.00th=[ 49], 50.00th=[ 68], 60.00th=[ 101],
| 70.00th=[ 148], 80.00th=[ 218], 90.00th=[ 342], 95.00th=[ 514],
| 99.00th=[ 5604], 99.50th=[ 7013], 99.90th=[ 8288], 99.95th=[ 8423],
| 99.99th=[ 8792]
bw ( MiB/s): min= 636, max= 1924, per=100.00%, avg=1177.84, stdev=51.23, samples=960
iops : min= 636, max= 1924, avg=1177.84, stdev=51.23, samples=960
lat (msec) : 10=0.12%, 20=10.82%, 50=30.09%, 100=18.94%, 250=23.61%
lat (msec) : 500=11.18%, 750=2.42%, 1000=0.68%, 2000=0.60%, >=2000=1.55%
cpu : usr=0.22%, sys=7.05%, ctx=133445, majf=0, minf=196978
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=141593,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=1178MiB/s (1235MB/s), 1178MiB/s-1178MiB/s (1235MB/s-1235MB/s), io=138GiB (148GB), run=120238-120238msec
```
```
job1: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
Jobs: 4 (f=4): [R(4)][100.0%][r=1183MiB/s][r=1183 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=4): err= 0: pid=132689: Sun Feb 9 12:56:32 2025
read: IOPS=1117, BW=1118MiB/s (1172MB/s)(131GiB/120339msec)
slat (usec): min=54, max=195553, avg=421.97, stdev=4088.77
clat (msec): min=7, max=11231, avg=228.48, stdev=732.36
lat (msec): min=7, max=11232, avg=228.90, stdev=732.80
clat percentiles (msec):
| 1.00th=[ 14], 5.00th=[ 20], 10.00th=[ 26], 20.00th=[ 38],
| 30.00th=[ 52], 40.00th=[ 71], 50.00th=[ 95], 60.00th=[ 129],
| 70.00th=[ 174], 80.00th=[ 239], 90.00th=[ 368], 95.00th=[ 523],
| 99.00th=[ 3641], 99.50th=[ 7215], 99.90th=[ 8356], 99.95th=[ 9060],
| 99.99th=[10805]
bw ( MiB/s): min= 52, max= 1770, per=100.00%, avg=1119.05, stdev=72.44, samples=960
iops : min= 52, max= 1770, avg=1119.05, stdev=72.44, samples=960
lat (msec) : 10=0.07%, 20=5.59%, 50=23.27%, 100=22.69%, 250=29.85%
lat (msec) : 500=13.05%, 750=2.73%, 1000=0.81%, 2000=0.61%, >=2000=1.33%
cpu : usr=0.30%, sys=6.92%, ctx=125102, majf=0, minf=232459
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=134538,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=1118MiB/s (1172MB/s), 1118MiB/s-1118MiB/s (1172MB/s-1172MB/s), io=131GiB (141GB), run=120339-120339msec
```
### Case 5 - seq read 28 processes
Jobfile
```
[global]
bs=256K
iodepth=64
direct=1
ioengine=libaio
group_reporting
time_based
runtime=120
numjobs=28
name=raw-read
rw=read
size=1G
[job1]
filename=/scratch/users/k2256813/benchmark/test1
```
Output
```
job1: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 28 processes
Jobs: 28 (f=28): [R(28)][100.0%][r=1179MiB/s][r=4716 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=28): err= 0: pid=234528: Sun Feb 9 13:00:30 2025
read: IOPS=4706, BW=1177MiB/s (1234MB/s)(138GiB/120485msec)
slat (usec): min=54, max=5704, avg=181.58, stdev=51.82
clat (msec): min=2, max=13460, avg=380.00, stdev=1282.10
lat (msec): min=2, max=13460, avg=380.19, stdev=1282.10
clat percentiles (msec):
| 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 9], 20.00th=[ 14],
| 30.00th=[ 21], 40.00th=[ 31], 50.00th=[ 48], 60.00th=[ 80],
| 70.00th=[ 136], 80.00th=[ 243], 90.00th=[ 558], 95.00th=[ 1720],
| 99.00th=[ 7752], 99.50th=[10000], 99.90th=[12550], 99.95th=[12818],
| 99.99th=[13221]
bw ( MiB/s): min= 20, max= 4566, per=100.00%, avg=1362.19, stdev=33.55, samples=5809
iops : min= 80, max=18265, avg=5448.75, stdev=134.19, samples=5809
lat (msec) : 4=0.25%, 10=13.40%, 20=16.10%, 50=21.03%, 100=13.63%
lat (msec) : 250=16.04%, 500=8.66%, 750=2.80%, 1000=1.35%, 2000=2.08%
lat (msec) : >=2000=4.65%
cpu : usr=0.13%, sys=3.26%, ctx=544521, majf=0, minf=230030
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.7%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=567111,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=1177MiB/s (1234MB/s), 1177MiB/s-1177MiB/s (1234MB/s-1234MB/s), io=138GiB (149GB), run=120485-120485msec
```
### Case 6 - seq read 1 process
This case shows that we can't saturate the bandwidth with 1 fio process
Input
```
[global]
bs=256K
iodepth=64
direct=1
ioengine=libaio
group_reporting
time_based
runtime=120
numjobs=1
name=raw-read
rw=read
size=1G
[job1]
filename=/scratch/users/k2256813/benchmark/test1
```
Output
```
job1: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=760MiB/s][r=3039 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=1): err= 0: pid=234628: Sun Feb 9 13:03:17 2025
read: IOPS=3386, BW=847MiB/s (888MB/s)(99.4GiB/120211msec)
slat (usec): min=51, max=4843, avg=165.12, stdev=124.09
clat (usec): min=1823, max=1115.6k, avg=18728.88, stdev=36821.54
lat (usec): min=1993, max=1115.7k, avg=18894.20, stdev=36813.90
clat percentiles (msec):
| 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7],
| 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 11],
| 70.00th=[ 13], 80.00th=[ 18], 90.00th=[ 36], 95.00th=[ 68],
| 99.00th=[ 163], 99.50th=[ 222], 99.90th=[ 477], 99.95th=[ 617],
| 99.99th=[ 827]
bw ( KiB/s): min=562688, max=1082880, per=100.00%, avg=868390.40, stdev=99229.00, samples=240
iops : min= 2198, max= 4230, avg=3392.15, stdev=387.61, samples=240
lat (msec) : 2=0.01%, 4=1.98%, 10=52.14%, 20=28.96%, 50=9.76%
lat (msec) : 100=4.44%, 250=2.31%, 500=0.31%, 750=0.07%, 1000=0.02%
lat (msec) : 2000=0.01%
cpu : usr=1.65%, sys=55.86%, ctx=233397, majf=0, minf=17998
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=407121,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=847MiB/s (888MB/s), 847MiB/s-847MiB/s (888MB/s-888MB/s), io=99.4GiB (107GB), run=120211-120211msec
```
With 2 output - around 13:05
```
job1: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 2 processes
Jobs: 1 (f=1): [R(1),_(1)][100.0%][eta 00m:00s]
job1: (groupid=0, jobs=2): err= 0: pid=132833: Sun Feb 9 13:05:21 2025
read: IOPS=2780, BW=695MiB/s (729MB/s)(85.6GiB/126138msec)
slat (usec): min=38, max=7102, avg=160.43, stdev=93.72
clat (usec): min=655, max=15486k, avg=45376.51, stdev=546472.40
lat (usec): min=1604, max=15486k, avg=45537.18, stdev=546470.37
clat percentiles (msec):
| 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6],
| 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 9], 60.00th=[ 11],
| 70.00th=[ 15], 80.00th=[ 23], 90.00th=[ 43], 95.00th=[ 74],
| 99.00th=[ 184], 99.50th=[ 292], 99.90th=[12953], 99.95th=[13758],
| 99.99th=[14563]
bw ( KiB/s): min=26624, max=1438208, per=100.00%, avg=765534.15, stdev=176710.53, samples=469
iops : min= 104, max= 5618, avg=2990.36, stdev=690.27, samples=469
lat (usec) : 750=0.01%
lat (msec) : 2=0.18%, 4=9.27%, 10=48.65%, 20=19.58%, 50=13.97%
lat (msec) : 100=5.34%, 250=2.40%, 500=0.19%, 750=0.02%, 1000=0.05%
lat (msec) : 2000=0.10%, >=2000=0.25%
cpu : usr=0.64%, sys=23.28%, ctx=262203, majf=0, minf=24740
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=350722,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=695MiB/s (729MB/s), 695MiB/s-695MiB/s (729MB/s-729MB/s), io=85.6GiB (91.9GB), run=126138-126138msec
```
Much slower than expected - why?
With 3 output - around 13:05
```
job1: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 3 processes
Jobs: 1 (f=1): [_(2),R(1)][100.0%][r=15.8MiB/s][r=63 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=3): err= 0: pid=234673: Sun Feb 9 13:05:46 2025
read: IOPS=3375, BW=844MiB/s (885MB/s)(102GiB/124242msec)
slat (usec): min=46, max=14479, avg=223.24, stdev=139.23
clat (usec): min=1650, max=15990k, avg=56248.34, stdev=614371.38
lat (usec): min=1810, max=15990k, avg=56471.82, stdev=614367.78
clat percentiles (msec):
| 1.00th=[ 3], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 7],
| 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 11], 60.00th=[ 13],
| 70.00th=[ 18], 80.00th=[ 27], 90.00th=[ 52], 95.00th=[ 83],
| 99.00th=[ 188], 99.50th=[ 305], 99.90th=[13087], 99.95th=[13758],
| 99.99th=[14966]
bw ( KiB/s): min=86528, max=1634304, per=100.00%, avg=901704.97, stdev=104746.15, samples=714
iops : min= 338, max= 6384, avg=3522.29, stdev=409.16, samples=714
lat (msec) : 2=0.06%, 4=4.43%, 10=45.39%, 20=23.98%, 50=15.91%
lat (msec) : 100=6.75%, 250=2.85%, 500=0.21%, 750=0.02%, 1000=0.01%
lat (msec) : 2000=0.01%, >=2000=0.39%
cpu : usr=0.76%, sys=25.90%, ctx=313501, majf=0, minf=31555
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=419344,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=844MiB/s (885MB/s), 844MiB/s-844MiB/s (885MB/s-885MB/s), io=102GiB (110GB), run=124242-124242msec
```
Much slower than expected - why?
Repeat of 2
```
job1: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 2 processes
Jobs: 2 (f=2): [R(2)][100.0%][r=1172MiB/s][r=4688 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=2): err= 0: pid=132885: Sun Feb 9 13:08:13 2025
read: IOPS=4611, BW=1153MiB/s (1209MB/s)(136GiB/120427msec)
slat (usec): min=44, max=8028, avg=143.60, stdev=47.43
clat (usec): min=1587, max=2149.6k, avg=27564.93, stdev=54307.82
lat (usec): min=1756, max=2149.7k, avg=27708.79, stdev=54306.66
clat percentiles (msec):
| 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 9],
| 30.00th=[ 11], 40.00th=[ 13], 50.00th=[ 16], 60.00th=[ 20],
| 70.00th=[ 26], 80.00th=[ 36], 90.00th=[ 56], 95.00th=[ 82],
| 99.00th=[ 163], 99.50th=[ 226], 99.90th=[ 827], 99.95th=[ 1301],
| 99.99th=[ 1703]
bw ( MiB/s): min= 538, max= 1485, per=100.00%, avg=1157.16, stdev=62.41, samples=479
iops : min= 2152, max= 5940, avg=4628.67, stdev=249.65, samples=479
lat (msec) : 2=0.01%, 4=0.94%, 10=27.63%, 20=31.60%, 50=28.04%
lat (msec) : 100=8.42%, 250=2.96%, 500=0.25%, 750=0.04%, 1000=0.04%
lat (msec) : 2000=0.07%, >=2000=0.01%
cpu : usr=1.09%, sys=34.48%, ctx=400565, majf=0, minf=52028
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=555316,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=1153MiB/s (1209MB/s), 1153MiB/s-1153MiB/s (1209MB/s-1209MB/s), io=136GiB (146GB), run=120427-120427msec
```
Repeat of 3
```
job1: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 3 processes
Jobs: 3 (f=3): [R(3)][100.0%][r=1178MiB/s][r=4710 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=3): err= 0: pid=234725: Sun Feb 9 13:08:05 2025
read: IOPS=4675, BW=1169MiB/s (1226MB/s)(137GiB/120073msec)
slat (usec): min=50, max=14698, avg=181.83, stdev=67.89
clat (usec): min=1839, max=2187.6k, avg=40870.42, stdev=63044.02
lat (msec): min=2, max=2187, avg=41.05, stdev=63.04
clat percentiles (msec):
| 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 9], 20.00th=[ 12],
| 30.00th=[ 16], 40.00th=[ 21], 50.00th=[ 27], 60.00th=[ 34],
| 70.00th=[ 44], 80.00th=[ 59], 90.00th=[ 86], 95.00th=[ 115],
| 99.00th=[ 203], 99.50th=[ 262], 99.90th=[ 902], 99.95th=[ 1435],
| 99.99th=[ 2022]
bw ( MiB/s): min= 694, max= 1596, per=100.00%, avg=1169.26, stdev=44.76, samples=720
iops : min= 2776, max= 6386, avg=4677.03, stdev=179.04, samples=720
lat (msec) : 2=0.01%, 4=0.17%, 10=14.21%, 20=25.23%, 50=35.24%
lat (msec) : 100=18.10%, 250=6.49%, 500=0.38%, 750=0.06%, 1000=0.03%
lat (msec) : 2000=0.08%, >=2000=0.01%
cpu : usr=1.02%, sys=29.60%, ctx=433163, majf=0, minf=35168
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=561432,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=1169MiB/s (1226MB/s), 1169MiB/s-1169MiB/s (1226MB/s-1226MB/s), io=137GiB (147GB), run=120073-120073msec
```
Repeat of 1 - again doesn't saturate, seems to be genuine
```
job1: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=813MiB/s][r=3252 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=1): err= 0: pid=234793: Sun Feb 9 13:10:38 2025
read: IOPS=3327, BW=832MiB/s (872MB/s)(97.6GiB/120131msec)
slat (usec): min=51, max=4484, avg=167.07, stdev=126.14
clat (usec): min=1378, max=1915.4k, avg=19060.43, stdev=51280.28
lat (usec): min=1831, max=1915.5k, avg=19227.67, stdev=51273.92
clat percentiles (msec):
| 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7],
| 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 11],
| 70.00th=[ 13], 80.00th=[ 17], 90.00th=[ 34], 95.00th=[ 65],
| 99.00th=[ 163], 99.50th=[ 224], 99.90th=[ 818], 99.95th=[ 1053],
| 99.99th=[ 1519]
bw ( KiB/s): min=466944, max=1054208, per=100.00%, avg=852744.53, stdev=110314.08, samples=240
iops : min= 1824, max= 4118, avg=3331.03, stdev=430.91, samples=240
lat (msec) : 2=0.02%, 4=2.54%, 10=52.92%, 20=28.42%, 50=9.31%
lat (msec) : 100=4.28%, 250=2.10%, 500=0.20%, 750=0.08%, 1000=0.06%
lat (msec) : 2000=0.06%
cpu : usr=1.67%, sys=55.49%, ctx=226708, majf=0, minf=4111
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=399787,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=832MiB/s (872MB/s), 832MiB/s-832MiB/s (872MB/s-872MB/s), io=97.6GiB (105GB), run=120131-120131msec
```
### Case 7 - random read latency
Jobfile
```
[global]
bs=4K
iodepth=1
direct=1
ioengine=libaio
group_reporting
time_based
runtime=120
numjobs=1
name=readlatency-test-job
rw=randread
[job1]
filename=/scratch/users/k2256813/benchmark/test1
```
Output
```
job1: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=284KiB/s][r=71 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=1): err= 0: pid=234847: Sun Feb 9 13:14:02 2025
read: IOPS=76, BW=306KiB/s (314kB/s)(35.9MiB/120002msec)
slat (usec): min=127, max=500, avg=169.64, stdev=17.16
clat (usec): min=731, max=613984, avg=12885.33, stdev=39086.79
lat (usec): min=881, max=614146, avg=13055.19, stdev=39087.48
clat percentiles (usec):
| 1.00th=[ 1020], 5.00th=[ 1123], 10.00th=[ 1188], 20.00th=[ 1287],
| 30.00th=[ 1369], 40.00th=[ 1532], 50.00th=[ 1729], 60.00th=[ 1975],
| 70.00th=[ 2311], 80.00th=[ 2933], 90.00th=[ 29492], 95.00th=[ 84411],
| 99.00th=[196084], 99.50th=[248513], 99.90th=[408945], 99.95th=[476054],
| 99.99th=[616563]
bw ( KiB/s): min= 8, max= 880, per=100.00%, avg=307.04, stdev=148.48, samples=239
iops : min= 2, max= 220, avg=76.76, stdev=37.12, samples=239
lat (usec) : 750=0.02%, 1000=0.77%
lat (msec) : 2=60.25%, 4=21.36%, 10=2.94%, 20=2.84%, 50=4.23%
lat (msec) : 100=3.71%, 250=3.38%, 500=0.47%, 750=0.02%
cpu : usr=0.07%, sys=1.36%, ctx=9221, majf=0, minf=13
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=9190,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=306KiB/s (314kB/s), 306KiB/s-306KiB/s (314kB/s-314kB/s), io=35.9MiB (37.6MB), run=120002-120002msec
```
### Case 8 - random read bandwidth
Jobfile
```
[global]
bs=256K
iodepth=64
direct=1
ioengine=libaio
group_reporting
time_based
runtime=120
numjobs=4
name=raw-randread
rw=randread
[job1]
filename=/scratch/users/k2256813/benchmark/test1
```
Output
```
job1: (g=0): rw=randread, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
Jobs: 2 (f=2): [r(1),_(1),r(1),_(1)][100.0%][r=167MiB/s][r=668 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=4): err= 0: pid=133014: Sun Feb 9 13:18:52 2025
read: IOPS=4657, BW=1164MiB/s (1221MB/s)(138GiB/120960msec)
slat (usec): min=59, max=8884, avg=174.61, stdev=33.64
clat (msec): min=2, max=1956, avg=54.69, stdev=106.58
lat (msec): min=2, max=1956, avg=54.87, stdev=106.58
clat percentiles (msec):
| 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 10],
| 30.00th=[ 11], 40.00th=[ 14], 50.00th=[ 17], 60.00th=[ 24],
| 70.00th=[ 37], 80.00th=[ 65], 90.00th=[ 142], 95.00th=[ 239],
| 99.00th=[ 531], 99.50th=[ 684], 99.90th=[ 1099], 99.95th=[ 1318],
| 99.99th=[ 1620]
bw ( MiB/s): min= 738, max= 1467, per=100.00%, avg=1173.21, stdev=26.20, samples=960
iops : min= 2954, max= 5868, avg=4692.84, stdev=104.79, samples=960
lat (msec) : 4=0.16%, 10=25.23%, 20=29.85%, 50=20.68%, 100=10.26%
lat (msec) : 250=9.30%, 500=3.37%, 750=0.76%, 1000=0.24%, 2000=0.15%
cpu : usr=1.02%, sys=21.61%, ctx=438328, majf=0, minf=35451
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=563393,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=1164MiB/s (1221MB/s), 1164MiB/s-1164MiB/s (1221MB/s-1221MB/s), io=138GiB (148GB), run=120960-120960msec
```
Output with 1 process
```
job1: (g=0): rw=randread, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=1094MiB/s][r=4375 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=1): err= 0: pid=234922: Sun Feb 9 13:20:28 2025
read: IOPS=4210, BW=1053MiB/s (1104MB/s)(124GiB/120179msec)
slat (usec): min=51, max=6953, avg=141.15, stdev=42.01
clat (usec): min=1724, max=1883.1k, avg=15056.63, stdev=41548.21
lat (usec): min=1864, max=1883.2k, avg=15197.93, stdev=41548.38
clat percentiles (msec):
| 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4],
| 30.00th=[ 5], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 6],
| 70.00th=[ 7], 80.00th=[ 10], 90.00th=[ 31], 95.00th=[ 69],
| 99.00th=[ 180], 99.50th=[ 230], 99.90th=[ 456], 99.95th=[ 651],
| 99.99th=[ 1284]
bw ( MiB/s): min= 764, max= 1157, per=100.00%, avg=1053.96, stdev=63.51, samples=240
iops : min= 3056, max= 4628, avg=4215.84, stdev=254.02, samples=240
lat (msec) : 2=0.03%, 4=27.40%, 10=53.84%, 20=6.18%, 50=5.57%
lat (msec) : 100=4.01%, 250=2.57%, 500=0.32%, 750=0.04%, 1000=0.02%
lat (msec) : 2000=0.02%
cpu : usr=2.08%, sys=59.57%, ctx=243449, majf=0, minf=19379
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=505964,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=1053MiB/s (1104MB/s), 1053MiB/s-1053MiB/s (1104MB/s-1104MB/s), io=124GiB (133GB), run=120179-120179msec
```
### Case 9 - random rw
Input
```
[global]
bs=256K
iodepth=64
direct=1
ioengine=libaio
group_reporting
time_based
runtime=120
numjobs=4
name=raw-randreadwrite
rw=randrw
size=1G
[job1]
filename=/scratch/users/k2256813/benchmark/test3
```
Output
```
job1: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
job1: Laying out IO file (1 file / 1024MiB)
Jobs: 4 (f=4): [m(4)][100.0%][eta 00m:00s]
job1: (groupid=0, jobs=4): err= 0: pid=235033: Sun Feb 9 13:26:10 2025
read: IOPS=1158, BW=290MiB/s (304MB/s)(36.7GiB/129904msec)
slat (usec): min=50, max=1969, avg=198.47, stdev=48.49
clat (usec): min=1717, max=15051k, avg=92206.70, stdev=458379.24
lat (usec): min=1899, max=15052k, avg=92405.50, stdev=458380.92
clat percentiles (msec):
| 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4],
| 30.00th=[ 5], 40.00th=[ 6], 50.00th=[ 10], 60.00th=[ 20],
| 70.00th=[ 36], 80.00th=[ 71], 90.00th=[ 174], 95.00th=[ 334],
| 99.00th=[ 1116], 99.50th=[ 2400], 99.90th=[ 6812], 99.95th=[ 9597],
| 99.99th=[12818]
bw ( KiB/s): min=159744, max=479744, per=100.00%, avg=320801.74, stdev=14656.32, samples=960
iops : min= 624, max= 1874, avg=1253.11, stdev=57.25, samples=960
write: IOPS=1159, BW=290MiB/s (304MB/s)(36.8GiB/129904msec); 0 zone resets
slat (usec): min=58, max=4829, avg=239.87, stdev=109.04
clat (msec): min=5, max=15012, avg=127.26, stdev=499.30
lat (msec): min=5, max=15013, avg=127.50, stdev=499.30
clat percentiles (msec):
| 1.00th=[ 10], 5.00th=[ 15], 10.00th=[ 18], 20.00th=[ 22],
| 30.00th=[ 27], 40.00th=[ 32], 50.00th=[ 39], 60.00th=[ 51],
| 70.00th=[ 72], 80.00th=[ 114], 90.00th=[ 226], 95.00th=[ 401],
| 99.00th=[ 1217], 99.50th=[ 2802], 99.90th=[ 9194], 99.95th=[ 9731],
| 99.99th=[12818]
bw ( KiB/s): min=169472, max=490496, per=100.00%, avg=321100.32, stdev=14166.73, samples=960
iops : min= 662, max= 1916, avg=1254.28, stdev=55.34, samples=960
lat (msec) : 2=0.07%, 4=14.69%, 10=10.89%, 20=12.91%, 50=28.83%
lat (msec) : 100=13.59%, 250=11.08%, 500=4.56%, 750=1.47%, 1000=0.66%
lat (msec) : 2000=0.66%, >=2000=0.57%
cpu : usr=1.23%, sys=12.33%, ctx=257127, majf=0, minf=16669
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=150473,150667,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=290MiB/s (304MB/s), 290MiB/s-290MiB/s (304MB/s-304MB/s), io=36.7GiB (39.4GB), run=129904-129904msec
WRITE: bw=290MiB/s (304MB/s), 290MiB/s-290MiB/s (304MB/s-304MB/s), io=36.8GiB (39.5GB), run=129904-129904msec
```
Does this show that performance of other benchmarks shows caching?
Test with a much larger file that won't be cached
### Case 10 - 100GB file seq read
Jobfile
```
[global]
bs=256K
iodepth=64
direct=1
ioengine=libaio
group_reporting
time_based
runtime=60
numjobs=4
name=raw-read
rw=read
size=100G
[job1]
filename=/scratch/users/k2256813/benchmark/test_100G
```
Output
```
job1: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
job1: Laying out IO file (1 file / 102400MiB)
Jobs: 4 (f=4): [R(4)][100.0%][r=585MiB/s][r=2341 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=4): err= 0: pid=235789: Sun Feb 9 13:33:08 2025
read: IOPS=2580, BW=645MiB/s (677MB/s)(38.0GiB/60326msec)
slat (usec): min=54, max=8994, avg=227.18, stdev=188.56
clat (usec): min=1661, max=1025.0k, avg=98924.95, stdev=136291.56
lat (usec): min=1833, max=1025.2k, avg=99152.32, stdev=136261.29
clat percentiles (msec):
| 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 7], 20.00th=[ 12],
| 30.00th=[ 19], 40.00th=[ 28], 50.00th=[ 42], 60.00th=[ 59],
| 70.00th=[ 85], 80.00th=[ 171], 90.00th=[ 296], 95.00th=[ 401],
| 99.00th=[ 600], 99.50th=[ 693], 99.90th=[ 877], 99.95th=[ 919],
| 99.99th=[ 1011]
bw ( KiB/s): min=250880, max=1191936, per=100.00%, avg=663210.67, stdev=41274.75, samples=480
iops : min= 980, max= 4656, avg=2590.68, stdev=161.23, samples=480
lat (msec) : 2=0.07%, 4=2.65%, 10=15.38%, 20=13.39%, 50=23.75%
lat (msec) : 100=17.65%, 250=14.05%, 500=10.69%, 750=2.09%, 1000=0.27%
lat (msec) : 2000=0.02%
cpu : usr=0.44%, sys=14.92%, ctx=124150, majf=0, minf=27026
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=155695,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=645MiB/s (677MB/s), 645MiB/s-645MiB/s (677MB/s-677MB/s), io=38.0GiB (40.8GB), run=60326-60326msec
```
Repeat after deleting file
```
job1: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
job1: Laying out IO file (1 file / 102400MiB)
Jobs: 4 (f=4): [R(4)][9.3%][r=771MiB/s][r=3085 IOPS][eta 09m:56s]
job1: (groupid=0, jobs=4): err= 0: pid=235900: Sun Feb 9 13:36:44 2025
read: IOPS=2531, BW=633MiB/s (664MB/s)(37.2GiB/60201msec)
slat (usec): min=53, max=7276, avg=224.25, stdev=174.54
clat (usec): min=1647, max=1528.2k, avg=100870.34, stdev=136508.59
lat (usec): min=1831, max=1528.3k, avg=101094.83, stdev=136480.13
clat percentiles (msec):
| 1.00th=[ 4], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 12],
| 30.00th=[ 21], 40.00th=[ 32], 50.00th=[ 48], 60.00th=[ 69],
| 70.00th=[ 101], 80.00th=[ 167], 90.00th=[ 275], 95.00th=[ 393],
| 99.00th=[ 609], 99.50th=[ 718], 99.90th=[ 978], 99.95th=[ 1385],
| 99.99th=[ 1519]
bw ( KiB/s): min=296960, max=1181184, per=100.00%, avg=649102.70, stdev=43387.20, samples=480
iops : min= 1160, max= 4614, avg=2535.53, stdev=169.47, samples=480
lat (msec) : 2=0.03%, 4=2.19%, 10=14.94%, 20=12.22%, 50=21.83%
lat (msec) : 100=18.77%, 250=18.27%, 500=9.51%, 750=1.78%, 1000=0.38%
lat (msec) : 2000=0.09%
cpu : usr=0.46%, sys=14.52%, ctx=123222, majf=0, minf=29709
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=152389,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=633MiB/s (664MB/s), 633MiB/s-633MiB/s (664MB/s-664MB/s), io=37.2GiB (39.9GB), run=60201-60201msec
```
Repeat keeping file
```
job1: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
Jobs: 4 (f=4): [R(4)][100.0%][r=658MiB/s][r=2632 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=4): err= 0: pid=235934: Sun Feb 9 13:37:53 2025
read: IOPS=3818, BW=955MiB/s (1001MB/s)(56.2GiB/60253msec)
slat (usec): min=52, max=14983, avg=199.21, stdev=113.11
clat (usec): min=1652, max=1425.2k, avg=66776.98, stdev=95989.52
lat (usec): min=1815, max=1425.4k, avg=66976.43, stdev=95978.41
clat percentiles (msec):
| 1.00th=[ 4], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 12],
| 30.00th=[ 17], 40.00th=[ 24], 50.00th=[ 33], 60.00th=[ 45],
| 70.00th=[ 64], 80.00th=[ 95], 90.00th=[ 161], 95.00th=[ 255],
| 99.00th=[ 477], 99.50th=[ 567], 99.90th=[ 810], 99.95th=[ 919],
| 99.99th=[ 995]
bw ( KiB/s): min=313344, max=1880064, per=100.00%, avg=980548.27, stdev=78527.08, samples=480
iops : min= 1224, max= 7344, avg=3830.27, stdev=306.75, samples=480
lat (msec) : 2=0.01%, 4=1.11%, 10=14.62%, 20=19.72%, 50=28.10%
lat (msec) : 100=17.62%, 250=13.63%, 500=4.39%, 750=0.64%, 1000=0.15%
lat (msec) : 2000=0.01%
cpu : usr=0.69%, sys=19.77%, ctx=186279, majf=0, minf=51174
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=230068,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=955MiB/s (1001MB/s), 955MiB/s-955MiB/s (1001MB/s-1001MB/s), io=56.2GiB (60.3GB), run=60253-60253msec
```
Repeat after deleting file
```
job1: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
job1: Laying out IO file (1 file / 102400MiB)
Jobs: 4 (f=4): [R(4)][100.0%][r=707MiB/s][r=2829 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=4): err= 0: pid=236216: Sun Feb 9 13:47:03 2025
read: IOPS=2707, BW=677MiB/s (710MB/s)(39.9GiB/60389msec)
slat (usec): min=48, max=7927, avg=225.54, stdev=174.79
clat (usec): min=1648, max=1234.9k, avg=94278.95, stdev=133825.06
lat (usec): min=1807, max=1235.1k, avg=94504.71, stdev=133797.42
clat percentiles (msec):
| 1.00th=[ 4], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 12],
| 30.00th=[ 20], 40.00th=[ 29], 50.00th=[ 42], 60.00th=[ 57],
| 70.00th=[ 81], 80.00th=[ 150], 90.00th=[ 275], 95.00th=[ 372],
| 99.00th=[ 634], 99.50th=[ 743], 99.90th=[ 969], 99.95th=[ 1028],
| 99.99th=[ 1200]
bw ( KiB/s): min=276791, max=1096192, per=100.00%, avg=696569.50, stdev=41634.45, samples=480
iops : min= 1081, max= 4282, avg=2720.97, stdev=162.64, samples=480
lat (msec) : 2=0.05%, 4=2.30%, 10=15.25%, 20=13.27%, 50=25.11%
lat (msec) : 100=18.11%, 250=14.54%, 500=9.09%, 750=1.82%, 1000=0.39%
lat (msec) : 2000=0.06%
cpu : usr=0.44%, sys=15.65%, ctx=131238, majf=0, minf=24737
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=163513,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=677MiB/s (710MB/s), 677MiB/s-677MiB/s (710MB/s-710MB/s), io=39.9GiB (42.9GB), run=60389-60389msec
```
### Case 11 - 100GB random read
Jobfile
```
[global]
bs=256K
iodepth=64
direct=1
ioengine=libaio
group_reporting
time_based
runtime=60
numjobs=4
name=raw-read
rw=randread
size=100G
[job1]
filename=/scratch/users/k2256813/benchmark/test_100G
```
Output
```
job1: (g=0): rw=randread, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
job1: Laying out IO file (1 file / 102400MiB)
Jobs: 4 (f=4): [r(4)][16.6%][r=1159MiB/s][r=4637 IOPS][eta 05m:06s]
job1: (groupid=0, jobs=4): err= 0: pid=236025: Sun Feb 9 13:42:50 2025
read: IOPS=4531, BW=1133MiB/s (1188MB/s)(66.8GiB/60391msec)
slat (usec): min=53, max=1090, avg=189.44, stdev=27.98
clat (msec): min=2, max=2202, avg=56.20, stdev=83.07
lat (msec): min=2, max=2203, avg=56.39, stdev=83.07
clat percentiles (msec):
| 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 9], 20.00th=[ 15],
| 30.00th=[ 20], 40.00th=[ 24], 50.00th=[ 29], 60.00th=[ 36],
| 70.00th=[ 50], 80.00th=[ 73], 90.00th=[ 130], 95.00th=[ 220],
| 99.00th=[ 397], 99.50th=[ 514], 99.90th=[ 793], 99.95th=[ 969],
| 99.99th=[ 1536]
bw ( MiB/s): min= 840, max= 1305, per=100.00%, avg=1139.16, stdev=25.40, samples=480
iops : min= 3360, max= 5222, avg=4556.63, stdev=101.60, samples=480
lat (msec) : 4=0.14%, 10=12.32%, 20=19.50%, 50=38.26%, 100=16.12%
lat (msec) : 250=10.28%, 500=2.84%, 750=0.42%, 1000=0.07%, 2000=0.05%
lat (msec) : >=2000=0.01%
cpu : usr=0.92%, sys=22.46%, ctx=218438, majf=0, minf=16697
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=273650,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=1133MiB/s (1188MB/s), 1133MiB/s-1133MiB/s (1188MB/s-1188MB/s), io=66.8GiB (71.7GB), run=60391-60391msec
```
This is faster than I expected - were the previous tests slowed down?
### Case 12 - random rw latency
Input
```
```
Output
```
job1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [m(1)][100.0%][r=24KiB/s,w=24KiB/s][r=6,w=6 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=1): err= 0: pid=236267: Sun Feb 9 13:51:17 2025
read: IOPS=14, BW=58.7KiB/s (60.1kB/s)(7048KiB/120075msec)
slat (usec): min=130, max=756, avg=173.90, stdev=26.19
clat (usec): min=891, max=417542, avg=17532.07, stdev=42407.83
lat (usec): min=1055, max=417716, avg=17706.18, stdev=42409.15
clat percentiles (usec):
| 1.00th=[ 1045], 5.00th=[ 1156], 10.00th=[ 1221], 20.00th=[ 1319],
| 30.00th=[ 1467], 40.00th=[ 1729], 50.00th=[ 2089], 60.00th=[ 2638],
| 70.00th=[ 8717], 80.00th=[ 17695], 90.00th=[ 47449], 95.00th=[ 89654],
| 99.00th=[233833], 99.50th=[287310], 99.90th=[367002], 99.95th=[417334],
| 99.99th=[417334]
bw ( KiB/s): min= 8, max= 200, per=100.00%, avg=61.29, stdev=38.82, samples=230
iops : min= 2, max= 50, avg=15.32, stdev= 9.71, samples=230
write: IOPS=15, BW=61.3KiB/s (62.7kB/s)(7356KiB/120075msec); 0 zone resets
slat (usec): min=152, max=109592, avg=236.55, stdev=2551.55
clat (msec): min=2, max=722, avg=48.09, stdev=86.36
lat (msec): min=2, max=722, avg=48.32, stdev=86.37
clat percentiles (msec):
| 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5],
| 30.00th=[ 6], 40.00th=[ 6], 50.00th=[ 8], 60.00th=[ 17],
| 70.00th=[ 29], 80.00th=[ 69], 90.00th=[ 163], 95.00th=[ 247],
| 99.00th=[ 393], 99.50th=[ 460], 99.90th=[ 592], 99.95th=[ 726],
| 99.99th=[ 726]
bw ( KiB/s): min= 8, max= 176, per=99.57%, avg=61.52, stdev=34.02, samples=239
iops : min= 2, max= 44, avg=15.38, stdev= 8.50, samples=239
lat (usec) : 1000=0.14%
lat (msec) : 2=23.30%, 4=16.36%, 10=22.02%, 20=10.91%, 50=10.64%
lat (msec) : 100=6.55%, 250=7.25%, 500=2.64%, 750=0.19%
cpu : usr=0.02%, sys=0.56%, ctx=3603, majf=0, minf=15
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1762,1839,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=58.7KiB/s (60.1kB/s), 58.7KiB/s-58.7KiB/s (60.1kB/s-60.1kB/s), io=7048KiB (7217kB), run=120075-120075msec
WRITE: bw=61.3KiB/s (62.7kB/s), 61.3KiB/s-61.3KiB/s (62.7kB/s-62.7kB/s), io=7356KiB (7533kB), run=120075-120075msec
```
### Case 13 -
Input
```
[global]
bs=256K
iodepth=64
direct=1
ioengine=libaio
group_reporting
time_based
runtime=120
numjobs=4
name=raw-write
rw=write
size=1G
[job1]
filename=/scratch/users/k2256813/benchmark/test6
```
Output
```
job1: (g=0): rw=write, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.28
Starting 4 processes
job1: Laying out IO file (1 file / 1024MiB)
job1: Laying out IO file (1 file / 1024MiB)
job1: Laying out IO file (1 file / 1024MiB)
job1: Laying out IO file (1 file / 1024MiB)
Jobs: 4 (f=4): [W(4)][100.0%][w=507MiB/s][w=2028 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=4): err= 0: pid=134086: Sun Feb 9 13:53:25 2025
write: IOPS=1976, BW=494MiB/s (518MB/s)(58.0GiB/120240msec); 0 zone resets
slat (usec): min=50, max=14511, avg=228.18, stdev=388.04
clat (msec): min=3, max=1838, avg=129.21, stdev=112.81
lat (msec): min=3, max=1838, avg=129.44, stdev=112.81
clat percentiles (msec):
| 1.00th=[ 20], 5.00th=[ 33], 10.00th=[ 41], 20.00th=[ 55],
| 30.00th=[ 69], 40.00th=[ 83], 50.00th=[ 99], 60.00th=[ 116],
| 70.00th=[ 140], 80.00th=[ 178], 90.00th=[ 253], 95.00th=[ 334],
| 99.00th=[ 575], 99.50th=[ 693], 99.90th=[ 1062], 99.95th=[ 1217],
| 99.99th=[ 1519]
bw ( KiB/s): min=210432, max=797184, per=100.00%, avg=506612.46, stdev=25013.38, samples=960
iops : min= 822, max= 3114, avg=1978.83, stdev=97.73, samples=960
lat (msec) : 4=0.01%, 10=0.14%, 20=1.01%, 50=15.55%, 100=34.54%
lat (msec) : 250=38.47%, 500=8.77%, 750=1.15%, 1000=0.25%, 2000=0.12%
cpu : usr=1.47%, sys=9.46%, ctx=64617, majf=0, minf=39458
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=0,237694,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
WRITE: bw=494MiB/s (518MB/s), 494MiB/s-494MiB/s (518MB/s-518MB/s), io=58.0GiB (62.3GB), run=120240-120240msec
```
### Case 14 - random read latency on 100G file
Input
```
[global]
bs=4K
iodepth=1
direct=1
ioengine=libaio
group_reporting
time_based
runtime=120
numjobs=1
name=readlatency-test-job
rw=randread
size=100G
[job1]
filename=/scratch/users/k2256813/benchmark/test_100G
```
Output
```
job1: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.28
Starting 1 process
job1: Laying out IO file (1 file / 102400MiB)
Jobs: 1 (f=1): [r(1)][100.0%][r=148KiB/s][r=37 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=1): err= 0: pid=236406: Sun Feb 9 13:59:13 2025
read: IOPS=26, BW=105KiB/s (108kB/s)(12.3MiB/120086msec)
slat (usec): min=107, max=325, avg=178.93, stdev=27.32
clat (usec): min=1709, max=869548, avg=37807.73, stdev=51401.37
lat (usec): min=1871, max=869718, avg=37986.95, stdev=51402.18
clat percentiles (msec):
| 1.00th=[ 7], 5.00th=[ 12], 10.00th=[ 14], 20.00th=[ 16],
| 30.00th=[ 18], 40.00th=[ 20], 50.00th=[ 21], 60.00th=[ 23],
| 70.00th=[ 29], 80.00th=[ 43], 90.00th=[ 81], 95.00th=[ 128],
| 99.00th=[ 257], 99.50th=[ 338], 99.90th=[ 468], 99.95th=[ 709],
| 99.99th=[ 869]
bw ( KiB/s): min= 8, max= 200, per=100.00%, avg=106.22, stdev=35.73, samples=238
iops : min= 2, max= 50, avg=26.55, stdev= 8.93, samples=238
lat (msec) : 2=0.13%, 4=0.54%, 10=2.21%, 20=43.09%, 50=37.17%
lat (msec) : 100=9.36%, 250=6.42%, 500=0.98%, 750=0.06%, 1000=0.03%
cpu : usr=0.02%, sys=0.50%, ctx=3161, majf=0, minf=14
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3161,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=105KiB/s (108kB/s), 105KiB/s-105KiB/s (108kB/s-108kB/s), io=12.3MiB (12.9MB), run=120086-120086msec
```
Google Drive
Gist
Clipboard
Sign in with Wallet
