# CloudFerro
## from dev-explorer pod
#### 4k randrw test test
```
$ fio --name TEST --eta-newline=5s --filename=temp.file --rw=randrw \
--size=500m --io_size=10g --blocksize=4k --ioengine=libaio --fsync=1 \
--iodepth=1 --direct=1 --numjobs=1 --runtime=60 --group_reporting
[...]
read: IOPS=38, BW=154KiB/s (158kB/s)(9268KiB/60013msec)
slat (usec): min=107, max=4817, avg=1064.87, stdev=268.42
clat (nsec): min=1427, max=56401, avg=3626.08, stdev=2177.33
lat (usec): min=109, max=4821, avg=1069.57, stdev=268.69
write: IOPS=40, BW=162KiB/s (166kB/s)(9728KiB/60013msec); 0 zone resets
slat (usec): min=119, max=4444, avg=247.91, stdev=209.53
clat (nsec): min=1403, max=59360, avg=2789.82, stdev=2348.27
lat (usec): min=121, max=4449, avg=251.49, stdev=209.88
```
<details>
<summary>full output</summary>
```
# fio --name TEST --eta-newline=5s --filename=temp.file --rw=randrw \
--size=500m --io_size=10g --blocksize=4k --ioengine=libaio --fsync=1 \
--iodepth=1 --direct=1 --numjobs=1 --runtime=60 --group_reporting
TEST: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.25
Starting 1 process
TEST: Laying out IO file (1 file / 500MiB)
Jobs: 1 (f=1): [m(1)][13.1%][r=164KiB/s,w=164KiB/s][r=41,w=41 IOPS][eta 00m:53s]
Jobs: 1 (f=1): [m(1)][23.0%][r=184KiB/s,w=156KiB/s][r=46,w=39 IOPS][eta 00m:47s]
Jobs: 1 (f=1): [m(1)][32.8%][r=188KiB/s,w=156KiB/s][r=47,w=39 IOPS][eta 00m:41s]
Jobs: 1 (f=1): [m(1)][42.6%][r=148KiB/s,w=164KiB/s][r=37,w=41 IOPS][eta 00m:35s]
Jobs: 1 (f=1): [m(1)][50.8%][r=184KiB/s,w=156KiB/s][r=46,w=39 IOPS][eta 00m:30s]
Jobs: 1 (f=1): [m(1)][60.7%][r=168KiB/s,w=168KiB/s][r=42,w=42 IOPS][eta 00m:24s]
Jobs: 1 (f=1): [m(1)][70.5%][r=151KiB/s,w=171KiB/s][r=37,w=42 IOPS][eta 00m:18s]
Jobs: 1 (f=1): [m(1)][80.3%][r=144KiB/s,w=152KiB/s][r=36,w=38 IOPS][eta 00m:12s]
Jobs: 1 (f=1): [m(1)][90.2%][r=176KiB/s,w=168KiB/s][r=44,w=42 IOPS][eta 00m:06s]
Jobs: 1 (f=1): [m(1)][100.0%][r=140KiB/s,w=152KiB/s][r=35,w=38 IOPS][eta 00m:00s]
TEST: (groupid=0, jobs=1): err= 0: pid=1553: Wed Feb 21 10:14:48 2024
read: IOPS=38, BW=154KiB/s (158kB/s)(9268KiB/60013msec)
slat (usec): min=107, max=4817, avg=1064.87, stdev=268.42
clat (nsec): min=1427, max=56401, avg=3626.08, stdev=2177.33
lat (usec): min=109, max=4821, avg=1069.57, stdev=268.69
clat percentiles (nsec):
| 1.00th=[ 2064], 5.00th=[ 2864], 10.00th=[ 2992], 20.00th=[ 3152],
| 30.00th=[ 3216], 40.00th=[ 3248], 50.00th=[ 3312], 60.00th=[ 3344],
| 70.00th=[ 3376], 80.00th=[ 3440], 90.00th=[ 3856], 95.00th=[ 6048],
| 99.00th=[18048], 99.50th=[19840], 99.90th=[24704], 99.95th=[28800],
| 99.99th=[56576]
bw ( KiB/s): min= 64, max= 256, per=99.72%, avg=154.68, stdev=39.27, samples=119
iops : min= 16, max= 64, avg=38.66, stdev= 9.82, samples=119
write: IOPS=40, BW=162KiB/s (166kB/s)(9728KiB/60013msec); 0 zone resets
slat (usec): min=119, max=4444, avg=247.91, stdev=209.53
clat (nsec): min=1403, max=59360, avg=2789.82, stdev=2348.27
lat (usec): min=121, max=4449, avg=251.49, stdev=209.88
clat percentiles (nsec):
| 1.00th=[ 1544], 5.00th=[ 1656], 10.00th=[ 1752], 20.00th=[ 2128],
| 30.00th=[ 2288], 40.00th=[ 2384], 50.00th=[ 2448], 60.00th=[ 2512],
| 70.00th=[ 2640], 80.00th=[ 3088], 90.00th=[ 3408], 95.00th=[ 3856],
| 99.00th=[17024], 99.50th=[19328], 99.90th=[32128], 99.95th=[32128],
| 99.99th=[59136]
bw ( KiB/s): min= 128, max= 208, per=99.94%, avg=162.41, stdev=14.94, samples=119
iops : min= 32, max= 52, avg=40.60, stdev= 3.74, samples=119
lat (usec) : 2=9.18%, 4=84.40%, 10=5.14%, 20=0.91%, 50=0.34%
lat (usec) : 100=0.04%
fsync/fdatasync/sync_file_range:
sync (nsec): min=120, max=16557, avg=553.60, stdev=670.96
sync percentiles (nsec):
| 1.00th=[ 197], 5.00th=[ 245], 10.00th=[ 270], 20.00th=[ 362],
| 30.00th=[ 506], 40.00th=[ 524], 50.00th=[ 532], 60.00th=[ 540],
| 70.00th=[ 548], 80.00th=[ 572], 90.00th=[ 660], 95.00th=[ 956],
| 99.00th=[ 1400], 99.50th=[ 1768], 99.90th=[15808], 99.95th=[16320],
| 99.99th=[16512]
cpu : usr=0.20%, sys=0.46%, ctx=11563, majf=0, minf=15
IO depths : 1=199.9%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2317,2432,0,4746 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=154KiB/s (158kB/s), 154KiB/s-154KiB/s (158kB/s-158kB/s), io=9268KiB (9490kB), run=60013-60013msec
WRITE: bw=162KiB/s (166kB/s), 162KiB/s-162KiB/s (166kB/s-166kB/s), io=9728KiB (9961kB), run=60013-60013msec
```
</details>
#### 1024k read only test
```
fio --name TEST --eta-newline=5s --filename=temp.file \
--rw=read --size=500m --io_size=10g --blocksize=1024k \
--ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 \
--numjobs=1 --runtime=60 --group_reporting
[...]
read: IOPS=467, BW=467MiB/s (490MB/s)(10.0GiB/21915msec)
slat (usec): min=1389, max=20985, avg=2132.16, stdev=742.66
clat (usec): min=5, max=187434, avg=64166.10, stdev=21443.59
lat (usec): min=1710, max=194251, avg=66299.37, stdev=22011.08
```
<details>
<summary>full output</summary>
```
fio --name TEST --eta-newline=5s --filename=temp.file --rw=read --size=500m --io_size=10g --blocksize=1024k --ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=60 --group_reporting
TEST: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
fio-3.25
Starting 1 process
Jobs: 1 (f=1): [R(1)][25.9%][r=493MiB/s][r=493 IOPS][eta 00m:20s]
Jobs: 1 (f=1): [R(1)][56.5%][r=503MiB/s][r=502 IOPS][eta 00m:10s]
Jobs: 1 (f=1): [R(1)][86.4%][r=498MiB/s][r=498 IOPS][eta 00m:03s]
Jobs: 1 (f=1): [R(1)][100.0%][r=516MiB/s][r=515 IOPS][eta 00m:00s]
TEST: (groupid=0, jobs=1): err= 0: pid=1556: Wed Feb 21 11:56:41 2024
read: IOPS=467, BW=467MiB/s (490MB/s)(10.0GiB/21915msec)
slat (usec): min=1389, max=20985, avg=2132.16, stdev=742.66
clat (usec): min=5, max=187434, avg=64166.10, stdev=21443.59
lat (usec): min=1710, max=194251, avg=66299.37, stdev=22011.08
clat percentiles (msec):
| 1.00th=[ 10], 5.00th=[ 48], 10.00th=[ 59], 20.00th=[ 60],
| 30.00th=[ 61], 40.00th=[ 62], 50.00th=[ 62], 60.00th=[ 63],
| 70.00th=[ 64], 80.00th=[ 65], 90.00th=[ 67], 95.00th=[ 71],
| 99.00th=[ 155], 99.50th=[ 157], 99.90th=[ 182], 99.95th=[ 182],
| 99.99th=[ 188]
bw ( KiB/s): min=124928, max=575488, per=99.60%, avg=476540.49, stdev=103255.37, samples=43
iops : min= 122, max= 562, avg=465.35, stdev=100.82, samples=43
lat (usec) : 10=0.19%, 20=0.01%, 50=0.01%
lat (msec) : 2=0.11%, 4=0.21%, 10=0.55%, 20=1.00%, 50=3.11%
lat (msec) : 100=90.16%, 250=4.67%
cpu : usr=0.38%, sys=5.48%, ctx=10566, majf=0, minf=8205
IO depths : 1=0.2%, 2=0.4%, 4=0.8%, 8=1.6%, 16=3.3%, 32=93.6%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.0%, 32=0.2%, 64=0.0%, >=64=0.0%
issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=467MiB/s (490MB/s), 467MiB/s-467MiB/s (490MB/s-490MB/s), io=10.0GiB (10.7GB), run=21915-21915msec
```
</details>
## from database-cluster pod
#### 4k randrw test test
```
$ fio --name TEST --eta-newline=5s --filename=temp.file --rw=randrw \
--size=500m --io_size=10g --blocksize=4k --ioengine=libaio --fsync=1 \
--iodepth=1 --direct=1 --numjobs=1 --runtime=60 --group_reporting
[...]
read: IOPS=37, BW=148KiB/s (152kB/s)(8896KiB/60021msec)
slat (usec): min=115, max=5894, avg=1005.91, stdev=234.89
clat (nsec): min=1696, max=159360, avg=3387.23, stdev=3649.95
lat (usec): min=117, max=5899, avg=1011.13, stdev=236.61
write: IOPS=38, BW=156KiB/s (160kB/s)(9352KiB/60021msec)
slat (usec): min=121, max=1593, avg=203.09, stdev=74.26
clat (nsec): min=1561, max=155816, avg=2956.79, stdev=3536.41
lat (usec): min=124, max=1598, avg=207.03, stdev=75.27
```
<details>
<summary>full output</summary>
```
TEST: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.1
Starting 1 process
TEST: Laying out IO file (1 file / 500MiB)
Jobs: 1 (f=1): [m(1)][13.1%][r=100KiB/s,w=148KiB/s][r=25,w=37 IOPS][eta 00m:53s]
Jobs: 1 (f=1): [m(1)][23.0%][r=148KiB/s,w=168KiB/s][r=37,w=42 IOPS][eta 00m:47s]
Jobs: 1 (f=1): [m(1)][32.8%][r=156KiB/s,w=168KiB/s][r=39,w=42 IOPS][eta 00m:41s]
Jobs: 1 (f=1): [m(1)][42.6%][r=156KiB/s,w=168KiB/s][r=39,w=42 IOPS][eta 00m:35s]
Jobs: 1 (f=1): [m(1)][52.5%][r=172KiB/s,w=152KiB/s][r=43,w=38 IOPS][eta 00m:29s]
Jobs: 1 (f=1): [m(1)][62.3%][r=172KiB/s,w=168KiB/s][r=43,w=42 IOPS][eta 00m:23s]
Jobs: 1 (f=1): [m(1)][72.1%][r=196KiB/s,w=148KiB/s][r=49,w=37 IOPS][eta 00m:17s]
Jobs: 1 (f=1): [m(1)][82.0%][r=200KiB/s,w=156KiB/s][r=50,w=39 IOPS][eta 00m:11s]
Jobs: 1 (f=1): [m(1)][91.8%][r=168KiB/s,w=136KiB/s][r=42,w=34 IOPS][eta 00m:05s]
Jobs: 1 (f=1): [m(1)][100.0%][r=96KiB/s,w=136KiB/s][r=24,w=34 IOPS][eta 00m:00s]
TEST: (groupid=0, jobs=1): err= 0: pid=1558569: Thu Feb 22 13:54:40 2024
read: IOPS=37, BW=148KiB/s (152kB/s)(8896KiB/60021msec)
slat (usec): min=115, max=5894, avg=1005.91, stdev=234.89
clat (nsec): min=1696, max=159360, avg=3387.23, stdev=3649.95
lat (usec): min=117, max=5899, avg=1011.13, stdev=236.61
clat percentiles (usec):
| 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3],
| 30.00th=[ 3], 40.00th=[ 3], 50.00th=[ 4], 60.00th=[ 4],
| 70.00th=[ 4], 80.00th=[ 4], 90.00th=[ 4], 95.00th=[ 5],
| 99.00th=[ 8], 99.50th=[ 19], 99.90th=[ 24], 99.95th=[ 31],
| 99.99th=[ 159]
bw ( KiB/s): min= 64, max= 264, per=100.00%, avg=148.10, stdev=45.51, samples=120
iops : min= 16, max= 66, avg=36.90, stdev=11.38, samples=120
write: IOPS=38, BW=156KiB/s (160kB/s)(9352KiB/60021msec)
slat (usec): min=121, max=1593, avg=203.09, stdev=74.26
clat (nsec): min=1561, max=155816, avg=2956.79, stdev=3536.41
lat (usec): min=124, max=1598, avg=207.03, stdev=75.27
clat percentiles (nsec):
| 1.00th=[ 1832], 5.00th=[ 1976], 10.00th=[ 2128], 20.00th=[ 2416],
| 30.00th=[ 2512], 40.00th=[ 2608], 50.00th=[ 2704], 60.00th=[ 2832],
| 70.00th=[ 2960], 80.00th=[ 3152], 90.00th=[ 3440], 95.00th=[ 3696],
| 99.00th=[ 5152], 99.50th=[ 17280], 99.90th=[ 20096], 99.95th=[ 42240],
| 99.99th=[156672]
bw ( KiB/s): min= 104, max= 200, per=100.00%, avg=155.70, stdev=19.43, samples=120
iops : min= 26, max= 50, avg=38.80, stdev= 4.91, samples=120
lat (usec) : 2=2.92%, 4=92.88%, 10=3.33%, 20=0.70%, 50=0.13%
lat (usec) : 250=0.04%
cpu : usr=0.18%, sys=0.50%, ctx=10481, majf=0, minf=8
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=2224,2338,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=148KiB/s (152kB/s), 148KiB/s-148KiB/s (152kB/s-152kB/s), io=8896KiB (9110kB), run=60021-60021msec
WRITE: bw=156KiB/s (160kB/s), 156KiB/s-156KiB/s (160kB/s-160kB/s), io=9352KiB (9576kB), run=60021-60021msec
```
</details>
# CCI2
## from master node
#### 4k randrw test
```
$ fio --name TEST --eta-newline=5s --filename=temp.file --rw=randrw \
--size=500m --io_size=10g --blocksize=4k --ioengine=libaio --fsync=1 \
--iodepth=1 --direct=1 --numjobs=1 --runtime=60 --group_reporting
[...]
read: IOPS=37, BW=149KiB/s (153kB/s)(8948KiB/60020msec)
slat (usec): min=11, max=235, avg=35.57, stdev=16.18
clat (usec): min=50, max=6479, avg=933.29, stdev=284.90
lat (usec): min=63, max=6535, avg=969.63, stdev=286.48
write: IOPS=39, BW=157KiB/s (161kB/s)(9416KiB/60020msec); 0 zone resets
slat (usec): min=19, max=779, avg=40.63, stdev=21.09
clat (usec): min=66, max=4534, avg=136.89, stdev=147.64
lat (usec): min=99, max=4573, avg=178.32, stdev=149.73
```
<details>
<summary>full output</summary>
```
fio --name TEST --eta-newline=5s --filename=temp.file --rw=randrw \
--size=500m --io_size=10g --blocksize=4k --ioengine=libaio --fsync=1 \
--iodepth=1 --direct=1 --numjobs=1 --runtime=60 --group_reporting
TEST: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.16
Starting 1 process
TEST: Laying out IO file (1 file / 400MiB)
Jobs: 1 (f=1): [m(1)][13.1%][r=184KiB/s,w=164KiB/s][r=46,w=41 IOPS][eta 00m:53s]
Jobs: 1 (f=1): [m(1)][23.0%][r=196KiB/s,w=160KiB/s][r=49,w=40 IOPS][eta 00m:47s]
Jobs: 1 (f=1): [m(1)][32.8%][r=184KiB/s,w=148KiB/s][r=46,w=37 IOPS][eta 00m:41s]
Jobs: 1 (f=1): [m(1)][42.6%][r=132KiB/s,w=164KiB/s][r=33,w=41 IOPS][eta 00m:35s]
Jobs: 1 (f=1): [m(1)][52.5%][r=188KiB/s,w=160KiB/s][r=47,w=40 IOPS][eta 00m:29s]
Jobs: 1 (f=1): [m(1)][62.3%][r=160KiB/s,w=152KiB/s][r=40,w=38 IOPS][eta 00m:23s]
Jobs: 1 (f=1): [m(1)][72.1%][r=140KiB/s,w=164KiB/s][r=35,w=41 IOPS][eta 00m:17s]
Jobs: 1 (f=1): [m(1)][82.0%][r=156KiB/s,w=156KiB/s][r=39,w=39 IOPS][eta 00m:11s]
Jobs: 1 (f=1): [m(1)][91.8%][r=164KiB/s,w=128KiB/s][r=41,w=32 IOPS][eta 00m:05s]
Jobs: 1 (f=1): [m(1)][100.0%][r=112KiB/s,w=168KiB/s][r=28,w=42 IOPS][eta 00m:00s]
TEST: (groupid=0, jobs=1): err= 0: pid=979671: Wed Feb 21 10:07:52 2024
read: IOPS=37, BW=149KiB/s (153kB/s)(8948KiB/60020msec)
slat (usec): min=11, max=235, avg=35.57, stdev=16.18
clat (usec): min=50, max=6479, avg=933.29, stdev=284.90
lat (usec): min=63, max=6535, avg=969.63, stdev=286.48
clat percentiles (usec):
| 1.00th=[ 77], 5.00th=[ 660], 10.00th=[ 783], 20.00th=[ 848],
| 30.00th=[ 889], 40.00th=[ 922], 50.00th=[ 947], 60.00th=[ 979],
| 70.00th=[ 1012], 80.00th=[ 1057], 90.00th=[ 1123], 95.00th=[ 1188],
| 99.00th=[ 1336], 99.50th=[ 1483], 99.90th=[ 4228], 99.95th=[ 5014],
| 99.99th=[ 6456]
bw ( KiB/s): min= 64, max= 248, per=100.00%, avg=149.11, stdev=43.54, samples=120
iops : min= 16, max= 62, avg=37.26, stdev=10.89, samples=120
write: IOPS=39, BW=157KiB/s (161kB/s)(9416KiB/60020msec); 0 zone resets
slat (usec): min=19, max=779, avg=40.63, stdev=21.09
clat (usec): min=66, max=4534, avg=136.89, stdev=147.64
lat (usec): min=99, max=4573, avg=178.32, stdev=149.73
clat percentiles (usec):
| 1.00th=[ 83], 5.00th=[ 91], 10.00th=[ 96], 20.00th=[ 102],
| 30.00th=[ 111], 40.00th=[ 118], 50.00th=[ 124], 60.00th=[ 129],
| 70.00th=[ 137], 80.00th=[ 147], 90.00th=[ 163], 95.00th=[ 184],
| 99.00th=[ 412], 99.50th=[ 635], 99.90th=[ 2802], 99.95th=[ 3359],
| 99.99th=[ 4555]
bw ( KiB/s): min= 112, max= 184, per=100.00%, avg=156.91, stdev=17.38, samples=120
iops : min= 28, max= 46, avg=39.21, stdev= 4.34, samples=120
lat (usec) : 100=10.63%, 250=42.13%, 500=0.46%, 750=1.46%, 1000=28.64%
lat (msec) : 2=16.51%, 4=0.09%, 10=0.09%
fsync/fdatasync/sync_file_range:
sync (nsec): min=143, max=70365, avg=794.85, stdev=1465.69
sync percentiles (nsec):
| 1.00th=[ 197], 5.00th=[ 330], 10.00th=[ 362], 20.00th=[ 414],
| 30.00th=[ 482], 40.00th=[ 506], 50.00th=[ 564], 60.00th=[ 700],
| 70.00th=[ 900], 80.00th=[ 1096], 90.00th=[ 1256], 95.00th=[ 1432],
| 99.00th=[ 1864], 99.50th=[ 4448], 99.90th=[16320], 99.95th=[16512],
| 99.99th=[70144]
cpu : usr=0.14%, sys=0.44%, ctx=13789, majf=0, minf=16
IO depths : 1=199.9%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2237,2354,0,4588 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=149KiB/s (153kB/s), 149KiB/s-149KiB/s (153kB/s-153kB/s), io=8948KiB (9163kB), run=60020-60020msec
WRITE: bw=157KiB/s (161kB/s), 157KiB/s-157KiB/s (161kB/s-161kB/s), io=9416KiB (9642kB), run=60020-60020msec
Disk stats (read/write):
vda: ios=2236/10421, merge=0/3501, ticks=2089/57269, in_queue=47824, util=99.88%
```
</details>
#### 1024k read only test
```
$ fio --name TEST --eta-newline=5s --filename=temp.file \
--rw=read --size=500m --io_size=10g --blocksize=1024k \
--ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 \
--numjobs=1 --runtime=60 --group_reporting
[...]
read: IOPS=2185, BW=2185MiB/s (2291MB/s)(10.0GiB/4686msec)
slat (usec): min=24, max=1789, avg=53.99, stdev=80.70
clat (usec): min=879, max=251152, avg=14385.95, stdev=19743.81
lat (usec): min=937, max=251248, avg=14440.49, stdev=19744.72
```
<details>
<summary>full output</summary>
```
$ fio --name TEST --eta-newline=5s --filename=temp.file --rw=read --size=500m --io_size=10g --blocksize=1024k --ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=60 --group_reporting
TEST: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
fio-3.16
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=2299MiB/s][r=2299 IOPS][eta 00m:00s]
TEST: (groupid=0, jobs=1): err= 0: pid=980507: Wed Feb 21 12:05:54 2024
read: IOPS=2185, BW=2185MiB/s (2291MB/s)(10.0GiB/4686msec)
slat (usec): min=24, max=1789, avg=53.99, stdev=80.70
clat (usec): min=879, max=251152, avg=14385.95, stdev=19743.81
lat (usec): min=937, max=251248, avg=14440.49, stdev=19744.72
clat percentiles (msec):
| 1.00th=[ 3], 5.00th=[ 5], 10.00th=[ 7], 20.00th=[ 9],
| 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 13],
| 70.00th=[ 14], 80.00th=[ 15], 90.00th=[ 18], 95.00th=[ 23],
| 99.00th=[ 146], 99.50th=[ 192], 99.90th=[ 230], 99.95th=[ 251],
| 99.99th=[ 251]
bw ( MiB/s): min= 1774, max= 2668, per=99.48%, avg=2173.78, stdev=296.21, samples=9
iops : min= 1774, max= 2668, avg=2173.78, stdev=296.21, samples=9
lat (usec) : 1000=0.01%
lat (msec) : 2=0.14%, 4=2.70%, 10=29.80%, 20=60.35%, 50=5.17%
lat (msec) : 100=0.74%, 250=1.08%, 500=0.01%
cpu : usr=2.09%, sys=13.28%, ctx=9541, majf=0, minf=8204
IO depths : 1=0.2%, 2=0.4%, 4=0.8%, 8=1.6%, 16=3.3%, 32=93.6%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.0%, 32=0.2%, 64=0.0%, >=64=0.0%
issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=2185MiB/s (2291MB/s), 2185MiB/s-2185MiB/s (2291MB/s-2291MB/s), io=10.0GiB (10.7GB), run=4686-4686msec
Disk stats (read/write):
vda: ios=10166/35, merge=0/1, ticks=120695/202, in_queue=101072, util=98.04%
```
</details>
## from database-cluster
#### 4k randrw test
```
$ fio --name TEST --eta-newline=5s --filename=temp.file --rw=randrw \
--size=500m --io_size=10g --blocksize=4k --ioengine=libaio --fsync=1 \
--iodepth=1 --direct=1 --numjobs=1 --runtime=60 --group_reporting
[...]
read: IOPS=13, BW=54.5KiB/s (55.8kB/s)(3272KiB/60042msec)
slat (usec): min=10, max=318, avg=30.90, stdev=22.03
clat (usec): min=332, max=447433, avg=17217.89, stdev=28268.72
lat (usec): min=344, max=447451, avg=17249.26, stdev=28268.38
write: IOPS=14, BW=57.8KiB/s (59.1kB/s)(3468KiB/60042msec)
slat (usec): min=10, max=229, avg=34.06, stdev=24.61
clat (usec): min=42, max=834, avg=101.42, stdev=49.01
lat (usec): min=65, max=897, avg=135.94, stdev=60.27
```
<details>
<summary>full output</summary>
```
# fio --name TEST --eta-newline=5s --filename=temp.file --rw=randrw \
> --size=500m --io_size=10g --blocksize=4k --ioengine=libaio --fsync=1 \
> --iodepth=1 --direct=1 --numjobs=1 --runtime=60 --group_reporting
TEST: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.1
Starting 1 process
TEST: Laying out IO file (1 file / 500MiB)
Jobs: 1 (f=1): [m(1)][13.1%][r=84KiB/s,w=132KiB/s][r=21,w=33 IOPS][eta 00m:53s]
Jobs: 1 (f=1): [m(1)][23.0%][r=132KiB/s,w=144KiB/s][r=33,w=36 IOPS][eta 00m:47s]
Jobs: 1 (f=1): [m(1)][32.8%][r=124KiB/s,w=108KiB/s][r=31,w=27 IOPS][eta 00m:41s]
Jobs: 1 (f=1): [m(1)][42.6%][r=108KiB/s,w=152KiB/s][r=27,w=38 IOPS][eta 00m:35s]
Jobs: 1 (f=1): [m(1)][52.5%][r=4KiB/s,w=16KiB/s][r=1,w=4 IOPS][eta 00m:29s]
Jobs: 1 (f=1): [m(1)][62.3%][r=32KiB/s,w=28KiB/s][r=8,w=7 IOPS][eta 00m:23s]
Jobs: 1 (f=1): [m(1)][72.1%][r=44KiB/s,w=16KiB/s][r=11,w=4 IOPS][eta 00m:17s]
Jobs: 1 (f=1): [m(1)][82.0%][r=16KiB/s,w=20KiB/s][r=4,w=5 IOPS][eta 00m:11s]
Jobs: 1 (f=1): [m(1)][91.8%][r=72KiB/s,w=56KiB/s][r=18,w=14 IOPS][eta 00m:05s]
Jobs: 1 (f=1): [m(1)][100.0%][r=40KiB/s,w=60KiB/s][r=10,w=15 IOPS][eta 00m:00s]
TEST: (groupid=0, jobs=1): err= 0: pid=600258: Thu Feb 22 13:43:52 2024
read: IOPS=13, BW=54.5KiB/s (55.8kB/s)(3272KiB/60042msec)
slat (usec): min=10, max=318, avg=30.90, stdev=22.03
clat (usec): min=332, max=447433, avg=17217.89, stdev=28268.72
lat (usec): min=344, max=447451, avg=17249.26, stdev=28268.38
clat percentiles (usec):
| 1.00th=[ 433], 5.00th=[ 498], 10.00th=[ 545], 20.00th=[ 742],
| 30.00th=[ 5407], 40.00th=[ 9503], 50.00th=[ 11600], 60.00th=[ 13435],
| 70.00th=[ 15795], 80.00th=[ 21627], 90.00th=[ 36439], 95.00th=[ 57410],
| 99.00th=[135267], 99.50th=[162530], 99.90th=[446694], 99.95th=[446694],
| 99.99th=[446694]
bw ( KiB/s): min= 7, max= 192, per=100.00%, avg=57.79, stdev=46.70, samples=113
iops : min= 1, max= 48, avg=14.41, stdev=11.67, samples=113
write: IOPS=14, BW=57.8KiB/s (59.1kB/s)(3468KiB/60042msec)
slat (usec): min=10, max=229, avg=34.06, stdev=24.61
clat (usec): min=42, max=834, avg=101.42, stdev=49.01
lat (usec): min=65, max=897, avg=135.94, stdev=60.27
clat percentiles (usec):
| 1.00th=[ 54], 5.00th=[ 62], 10.00th=[ 68], 20.00th=[ 75],
| 30.00th=[ 82], 40.00th=[ 88], 50.00th=[ 94], 60.00th=[ 101],
| 70.00th=[ 109], 80.00th=[ 119], 90.00th=[ 141], 95.00th=[ 161],
| 99.00th=[ 212], 99.50th=[ 310], 99.90th=[ 832], 99.95th=[ 832],
| 99.99th=[ 832]
bw ( KiB/s): min= 7, max= 216, per=100.00%, avg=59.22, stdev=54.42, samples=117
iops : min= 1, max= 54, avg=14.76, stdev=13.62, samples=117
lat (usec) : 50=0.06%, 100=30.33%, 250=20.65%, 500=2.91%, 750=7.36%
lat (usec) : 1000=2.31%
lat (msec) : 2=1.13%, 4=0.36%, 10=6.94%, 20=17.15%, 50=7.54%
lat (msec) : 100=2.26%, 250=0.95%, 500=0.06%
cpu : usr=0.04%, sys=0.14%, ctx=4275, majf=0, minf=10
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=818,867,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=54.5KiB/s (55.8kB/s), 54.5KiB/s-54.5KiB/s (55.8kB/s-55.8kB/s), io=3272KiB (3351kB), run=60042-60042msec
WRITE: bw=57.8KiB/s (59.1kB/s), 57.8KiB/s-57.8KiB/s (59.1kB/s-59.1kB/s), io=3468KiB (3551kB), run=60042-60042msec
```
</details>
## from master node - SSD
#### 4k randrw test
```
$ fio --name TEST --eta-newline=5s --filename=temp.file --rw=randrw \
--size=500m --io_size=10g --blocksize=4k --ioengine=libaio --fsync=1 \
--iodepth=1 --direct=1 --numjobs=1 --runtime=60 --group_reporting
[...]
read: IOPS=567, BW=2270KiB/s (2325kB/s)(133MiB/60001msec)
slat (usec): min=5, max=2285, avg=14.30, stdev=15.75
clat (usec): min=2, max=4571, avg=318.91, stdev=93.54
lat (usec): min=143, max=4582, avg=333.35, stdev=95.06
write: IOPS=568, BW=2274KiB/s (2329kB/s)(133MiB/60001msec); 0 zone resets
slat (usec): min=5, max=927, avg=14.76, stdev= 8.28
clat (nsec): min=1052, max=70555k, avg=66902.94, stdev=383045.02
lat (usec): min=40, max=70565, avg=81.80, stdev=383.19
```
<details>
<summary>full output</summary>
```
fio --name TEST --eta-newline=5s --filename=temp.file --rw=randrw --size=2g --io_size=10g --blocksize=4k --ioengine=libaio --fsync=1 --iodepth=1 --direct=1 --numjobs=1 --runtime=60 --group_reporting
TEST: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [m(1)][11.7%][r=2306KiB/s,w=2210KiB/s][r=576,w=552 IOPS][eta 00m:53s]
Jobs: 1 (f=1): [m(1)][21.7%][r=2336KiB/s,w=2288KiB/s][r=584,w=572 IOPS][eta 00m:47s]
Jobs: 1 (f=1): [m(1)][31.7%][r=2182KiB/s,w=2122KiB/s][r=545,w=530 IOPS][eta 00m:41s]
Jobs: 1 (f=1): [m(1)][41.7%][r=2272KiB/s,w=2304KiB/s][r=568,w=576 IOPS][eta 00m:35s]
Jobs: 1 (f=1): [m(1)][51.7%][r=2268KiB/s,w=2360KiB/s][r=567,w=590 IOPS][eta 00m:29s]
Jobs: 1 (f=1): [m(1)][61.7%][r=2362KiB/s,w=2270KiB/s][r=590,w=567 IOPS][eta 00m:23s]
Jobs: 1 (f=1): [m(1)][71.7%][r=2148KiB/s,w=2284KiB/s][r=537,w=571 IOPS][eta 00m:17s]
Jobs: 1 (f=1): [m(1)][81.7%][r=2064KiB/s,w=2364KiB/s][r=516,w=591 IOPS][eta 00m:11s]
Jobs: 1 (f=1): [m(1)][91.7%][r=2284KiB/s,w=2288KiB/s][r=571,w=572 IOPS][eta 00m:05s]
Jobs: 1 (f=1): [m(1)][100.0%][r=2122KiB/s,w=2190KiB/s][r=530,w=547 IOPS][eta 00m:00s]
TEST: (groupid=0, jobs=1): err= 0: pid=582748: Thu Feb 22 09:48:34 2024
read: IOPS=567, BW=2270KiB/s (2325kB/s)(133MiB/60001msec)
slat (usec): min=5, max=2285, avg=14.30, stdev=15.75
clat (usec): min=2, max=4571, avg=318.91, stdev=93.54
lat (usec): min=143, max=4582, avg=333.35, stdev=95.06
clat percentiles (usec):
| 1.00th=[ 174], 5.00th=[ 200], 10.00th=[ 223], 20.00th=[ 273],
| 30.00th=[ 293], 40.00th=[ 306], 50.00th=[ 318], 60.00th=[ 330],
| 70.00th=[ 343], 80.00th=[ 359], 90.00th=[ 388], 95.00th=[ 424],
| 99.00th=[ 570], 99.50th=[ 652], 99.90th=[ 1106], 99.95th=[ 1745],
| 99.99th=[ 3294]
bw ( KiB/s): min= 1904, max= 2776, per=100.00%, avg=2271.15, stdev=167.61, samples=119
iops : min= 476, max= 694, avg=567.78, stdev=41.91, samples=119
write: IOPS=568, BW=2274KiB/s (2329kB/s)(133MiB/60001msec); 0 zone resets
slat (usec): min=5, max=927, avg=14.76, stdev= 8.28
clat (nsec): min=1052, max=70555k, avg=66902.94, stdev=383045.02
lat (usec): min=40, max=70565, avg=81.80, stdev=383.19
clat percentiles (usec):
| 1.00th=[ 39], 5.00th=[ 45], 10.00th=[ 48], 20.00th=[ 51],
| 30.00th=[ 55], 40.00th=[ 59], 50.00th=[ 62], 60.00th=[ 65],
| 70.00th=[ 69], 80.00th=[ 73], 90.00th=[ 82], 95.00th=[ 94],
| 99.00th=[ 145], 99.50th=[ 186], 99.90th=[ 334], 99.95th=[ 420],
| 99.99th=[ 832]
bw ( KiB/s): min= 1920, max= 2688, per=100.00%, avg=2276.46, stdev=143.75, samples=119
iops : min= 480, max= 672, avg=569.11, stdev=35.94, samples=119
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=8.59%
lat (usec) : 100=39.54%, 250=8.99%, 500=41.80%, 750=0.91%, 1000=0.07%
lat (msec) : 2=0.05%, 4=0.02%, 10=0.01%, 100=0.01%
fsync/fdatasync/sync_file_range:
sync (nsec): min=50, max=10210k, avg=346.93, stdev=39105.95
sync percentiles (nsec):
| 1.00th=[ 81], 5.00th=[ 100], 10.00th=[ 110], 20.00th=[ 120],
| 30.00th=[ 131], 40.00th=[ 151], 50.00th=[ 161], 60.00th=[ 181],
| 70.00th=[ 201], 80.00th=[ 251], 90.00th=[ 310], 95.00th=[ 382],
| 99.00th=[ 502], 99.50th=[ 580], 99.90th=[ 2192], 99.95th=[ 7264],
| 99.99th=[14400]
cpu : usr=0.81%, sys=2.72%, ctx=176222, majf=0, minf=20
IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=34051,34117,0,68165 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=2270KiB/s (2325kB/s), 2270KiB/s-2270KiB/s (2325kB/s-2325kB/s), io=133MiB (139MB), run=60001-60001msec
WRITE: bw=2274KiB/s (2329kB/s), 2274KiB/s-2274KiB/s (2329kB/s-2329kB/s), io=133MiB (140MB), run=60001-60001msec
Disk stats (read/write):
vda: ios=33988/133726, merge=0/29747, ticks=10962/46793, in_queue=97476, util=99.87%
```
</details>
#### 1024k read only test
```
$ fio --name TEST --eta-newline=5s --filename=temp.file \
--rw=read --size=500m --io_size=10g --blocksize=1024k \
--ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 \
--numjobs=1 --runtime=60 --group_reporting
[...]
read: IOPS=3923, BW=3923MiB/s (4114MB/s)(10.0GiB/2610msec)
slat (usec): min=21, max=576, avg=35.50, stdev=23.08
clat (usec): min=1038, max=45490, avg=8091.25, stdev=5350.83
lat (usec): min=1067, max=45541, avg=8127.05, stdev=5350.84
```
<details>
<summary>full output</summary>
```
fio --name TEST --eta-newline=5s --filename=temp.file --rw=read --size=2g --io_size=10g --blocksize=1024k --ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=60 --group_reporting
TEST: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
fio-3.28
Starting 1 process
TEST: Laying out IO file (1 file / 2048MiB)
Jobs: 1 (f=1): [R(1)][-.-%][r=3945MiB/s][r=3945 IOPS][eta 00m:00s]
TEST: (groupid=0, jobs=1): err= 0: pid=572342: Thu Feb 22 09:27:42 2024
read: IOPS=3923, BW=3923MiB/s (4114MB/s)(10.0GiB/2610msec)
slat (usec): min=21, max=576, avg=35.50, stdev=23.08
clat (usec): min=1038, max=45490, avg=8091.25, stdev=5350.83
lat (usec): min=1067, max=45541, avg=8127.05, stdev=5350.84
clat percentiles (usec):
| 1.00th=[ 1663], 5.00th=[ 2089], 10.00th=[ 2573], 20.00th=[ 3458],
| 30.00th=[ 4359], 40.00th=[ 5473], 50.00th=[ 6783], 60.00th=[ 8225],
| 70.00th=[ 9896], 80.00th=[12256], 90.00th=[15533], 95.00th=[18482],
| 99.00th=[24249], 99.50th=[28181], 99.90th=[34341], 99.95th=[38011],
| 99.99th=[43254]
bw ( MiB/s): min= 3678, max= 4258, per=100.00%, avg=3934.80, stdev=211.05, samples=5
iops : min= 3678, max= 4258, avg=3934.80, stdev=211.05, samples=5
lat (msec) : 2=4.19%, 4=21.63%, 10=44.33%, 20=26.53%, 50=3.32%
cpu : usr=1.61%, sys=15.37%, ctx=9285, majf=0, minf=8204
IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.4%, 16=0.8%, 32=98.5%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=3923MiB/s (4114MB/s), 3923MiB/s-3923MiB/s (4114MB/s-4114MB/s), io=10.0GiB (10.7GB), run=2610-2610msec
Disk stats (read/write):
vda: ios=30041/149, merge=0/31, ticks=239873/890, in_queue=241478, util=96.50%
```
</details>
# OVH
## from a pod
#### 4k randrw test
```
$ fio --name TEST --eta-newline=5s --filename=temp.file --rw=randrw \
--size=500m --io_size=10g --blocksize=4k --ioengine=libaio --fsync=1 \
--iodepth=1 --direct=1 --numjobs=1 --runtime=60 --group_reporting
[...]
read: IOPS=791, BW=3167KiB/s (3243kB/s)(186MiB/60001msec)
slat (usec): min=7, max=890, avg=34.88, stdev=24.16
clat (usec): min=6, max=13775, avg=219.72, stdev=139.66
lat (usec): min=109, max=13814, avg=254.60, stdev=146.79
write: IOPS=786, BW=3148KiB/s (3223kB/s)(184MiB/60001msec); 0 zone resets
slat (usec): min=7, max=1147, avg=36.34, stdev=24.94
clat (nsec): min=1979, max=14352k, avg=190479.97, stdev=158744.77
lat (usec): min=72, max=14368, avg=226.82, stdev=165.37
```
<details>
<summary>full output</summary>
```
$ fio --name TEST --eta-newline=5s --filename=temp.file --rw=randrw \
--size=500m --io_size=10g --blocksize=4k --ioengine=libaio --fsync=1 \
--iodepth=1 --direct=1 --numjobs=1 --runtime=60 --group_reporting
TEST: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [m(1)][11.7%][r=3151KiB/s,w=3015KiB/s][r=787,w=753 IOPS][eta 00m:53s]
Jobs: 1 (f=1): [m(1)][21.7%][r=3984KiB/s,w=3860KiB/s][r=996,w=965 IOPS][eta 00m:47s]
Jobs: 1 (f=1): [m(1)][31.7%][r=2932KiB/s,w=2936KiB/s][r=733,w=734 IOPS][eta 00m:41s]
Jobs: 1 (f=1): [m(1)][41.7%][r=3448KiB/s,w=3652KiB/s][r=862,w=913 IOPS][eta 00m:35s]
Jobs: 1 (f=1): [m(1)][51.7%][r=3072KiB/s,w=2784KiB/s][r=768,w=696 IOPS][eta 00m:29s]
Jobs: 1 (f=1): [m(1)][61.7%][r=3644KiB/s,w=3784KiB/s][r=911,w=946 IOPS][eta 00m:23s]
Jobs: 1 (f=1): [m(1)][71.7%][r=3032KiB/s,w=3164KiB/s][r=758,w=791 IOPS][eta 00m:17s]
Jobs: 1 (f=1): [m(1)][81.7%][r=2510KiB/s,w=2526KiB/s][r=627,w=631 IOPS][eta 00m:11s]
Jobs: 1 (f=1): [m(1)][90.2%][r=4208KiB/s,w=3900KiB/s][r=1052,w=975 IOPS][eta 00m:06s]
Jobs: 1 (f=1): [m(1)][100.0%][r=3975KiB/s,w=3919KiB/s][r=993,w=979 IOPS][eta 00m:00s]
TEST: (groupid=0, jobs=1): err= 0: pid=5181: Wed Feb 21 14:00:42 2024
read: IOPS=791, BW=3167KiB/s (3243kB/s)(186MiB/60001msec)
slat (usec): min=7, max=890, avg=34.88, stdev=24.16
clat (usec): min=6, max=13775, avg=219.72, stdev=139.66
lat (usec): min=109, max=13814, avg=254.60, stdev=146.79
clat percentiles (usec):
| 1.00th=[ 135], 5.00th=[ 149], 10.00th=[ 157], 20.00th=[ 167],
| 30.00th=[ 180], 40.00th=[ 194], 50.00th=[ 206], 60.00th=[ 219],
| 70.00th=[ 231], 80.00th=[ 245], 90.00th=[ 269], 95.00th=[ 306],
| 99.00th=[ 627], 99.50th=[ 742], 99.90th=[ 1106], 99.95th=[ 1319],
| 99.99th=[ 4490]
bw ( KiB/s): min= 2208, max= 6280, per=100.00%, avg=3174.24, stdev=905.25, samples=119
iops : min= 552, max= 1570, avg=793.55, stdev=226.31, samples=119
write: IOPS=786, BW=3148KiB/s (3223kB/s)(184MiB/60001msec); 0 zone resets
slat (usec): min=7, max=1147, avg=36.34, stdev=24.94
clat (nsec): min=1979, max=14352k, avg=190479.97, stdev=158744.77
lat (usec): min=72, max=14368, avg=226.82, stdev=165.37
clat percentiles (usec):
| 1.00th=[ 102], 5.00th=[ 116], 10.00th=[ 126], 20.00th=[ 137],
| 30.00th=[ 149], 40.00th=[ 167], 50.00th=[ 182], 60.00th=[ 194],
| 70.00th=[ 206], 80.00th=[ 225], 90.00th=[ 251], 95.00th=[ 281],
| 99.00th=[ 478], 99.50th=[ 627], 99.90th=[ 1020], 99.95th=[ 1401],
| 99.99th=[ 8979]
bw ( KiB/s): min= 2136, max= 6048, per=100.00%, avg=3152.46, stdev=864.44, samples=119
iops : min= 534, max= 1512, avg=788.11, stdev=216.11, samples=119
lat (usec) : 2=0.01%, 4=0.01%, 10=0.02%, 20=0.01%, 50=0.01%
lat (usec) : 100=0.35%, 250=86.02%, 500=12.05%, 750=1.19%, 1000=0.24%
lat (msec) : 2=0.10%, 4=0.01%, 10=0.01%, 20=0.01%
fsync/fdatasync/sync_file_range:
sync (usec): min=120, max=15030, avg=594.10, stdev=473.85
sync percentiles (usec):
| 1.00th=[ 196], 5.00th=[ 217], 10.00th=[ 231], 20.00th=[ 258],
| 30.00th=[ 302], 40.00th=[ 351], 50.00th=[ 392], 60.00th=[ 453],
| 70.00th=[ 750], 80.00th=[ 1057], 90.00th=[ 1237], 95.00th=[ 1336],
| 99.00th=[ 1598], 99.50th=[ 1778], 99.90th=[ 2606], 99.95th=[ 4883],
| 99.99th=[14484]
cpu : usr=2.62%, sys=10.06%, ctx=258205, majf=0, minf=14
IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=47503,47219,0,94718 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=3167KiB/s (3243kB/s), 3167KiB/s-3167KiB/s (3243kB/s-3243kB/s), io=186MiB (195MB), run=60001-60001msec
WRITE: bw=3148KiB/s (3223kB/s), 3148KiB/s-3148KiB/s (3223kB/s-3223kB/s), io=184MiB (193MB), run=60001-60001msec
```
</details>
#### 1024 read only test
```
$ fio --name TEST --eta-newline=5s --filename=temp.file \
--rw=read --size=500m --io_size=10g --blocksize=1024k \
--ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 \
--numjobs=1 --runtime=60 --group_reporting
[...]
read: IOPS=2442, BW=2443MiB/s (2561MB/s)(10.0GiB/4192msec)
slat (usec): min=39, max=1583, avg=149.53, stdev=82.49
clat (usec): min=832, max=50823, avg=12705.42, stdev=4936.54
lat (usec): min=972, max=50974, avg=12854.95, stdev=4940.71
```
<details>
<summary>full output</summary>
```
$ fio --name TEST --eta-newline=5s --filename=temp.file \
--rw=read --size=500m --io_size=10g --blocksize=1024k \
--ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 \
--numjobs=1 --runtime=60 --group_reporting
TEST: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=2441MiB/s][r=2441 IOPS][eta 00m:00s]
TEST: (groupid=0, jobs=1): err= 0: pid=5184: Wed Feb 21 14:01:57 2024
read: IOPS=2442, BW=2443MiB/s (2561MB/s)(10.0GiB/4192msec)
slat (usec): min=39, max=1583, avg=149.53, stdev=82.49
clat (usec): min=832, max=50823, avg=12705.42, stdev=4936.54
lat (usec): min=972, max=50974, avg=12854.95, stdev=4940.71
clat percentiles (usec):
| 1.00th=[ 1467], 5.00th=[ 3294], 10.00th=[ 4817], 20.00th=[ 7767],
| 30.00th=[10421], 40.00th=[13173], 50.00th=[15401], 60.00th=[15664],
| 70.00th=[15795], 80.00th=[15926], 90.00th=[16450], 95.00th=[17957],
| 99.00th=[21890], 99.50th=[24511], 99.90th=[30540], 99.95th=[30802],
| 99.99th=[50594]
bw ( MiB/s): min= 2240, max= 2550, per=99.94%, avg=2441.25, stdev=97.56, samples=8
iops : min= 2240, max= 2550, avg=2441.25, stdev=97.56, samples=8
lat (usec) : 1000=0.22%
lat (msec) : 2=1.93%, 4=4.58%, 10=21.68%, 20=68.79%, 50=2.76%
lat (msec) : 100=0.03%
cpu : usr=2.12%, sys=38.80%, ctx=6386, majf=0, minf=8202
IO depths : 1=0.2%, 2=0.4%, 4=0.8%, 8=1.6%, 16=3.3%, 32=93.6%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.0%, 32=0.2%, 64=0.0%, >=64=0.0%
issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=2443MiB/s (2561MB/s), 2443MiB/s-2443MiB/s (2561MB/s-2561MB/s), io=10.0GiB (10.7GB), run=4192-4192msec
```
</details>
# Raspberry PI 5 on SSD
#### 4k randrw test
```
$ fio --name TEST --eta-newline=5s --filename=temp.file --rw=randrw \
--size=500m --io_size=10g --blocksize=4k --ioengine=libaio --fsync=1 \
--iodepth=1 --direct=1 --numjobs=1 --runtime=60 --group_reporting
[...]
read: IOPS=527, BW=2110KiB/s (2160kB/s)(124MiB/60001msec)
slat (usec): min=9, max=464, avg=19.47, stdev= 6.89
clat (usec): min=3, max=16846, avg=180.23, stdev=132.30
lat (usec): min=141, max=16866, avg=199.70, stdev=132.51
write: IOPS=528, BW=2113KiB/s (2164kB/s)(124MiB/60001msec); 0 zone resets
slat (usec): min=10, max=695, avg=21.82, stdev=10.01
clat (nsec): min=1648, max=7468.6k, avg=102478.22, stdev=52280.64
lat (usec): min=80, max=7494, avg=124.30, stdev=55.32
```
<details>
<summary>full output</summary>
```
$ fio --name TEST --eta-newline=5s --filename=temp.file --rw=randrw \
--size=500m --io_size=10g --blocksize=4k --ioengine=libaio --fsync=1 \
--iodepth=1 --direct=1 --numjobs=1 --runtime=60 --group_reporting
TEST: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.33
Starting 1 process
TEST: Laying out IO file (1 file / 500MiB)
Jobs: 1 (f=1): [m(1)][11.7%][r=2134KiB/s,w=2062KiB/s][r=533,w=515 IOPS][eta 00m:53s]
Jobs: 1 (f=1): [m(1)][21.7%][r=2098KiB/s,w=2074KiB/s][r=524,w=518 IOPS][eta 00m:47s]
Jobs: 1 (f=1): [m(1)][31.7%][r=2076KiB/s,w=2096KiB/s][r=519,w=524 IOPS][eta 00m:41s]
Jobs: 1 (f=1): [m(1)][41.7%][r=1956KiB/s,w=2160KiB/s][r=489,w=540 IOPS][eta 00m:35s]
Jobs: 1 (f=1): [m(1)][51.7%][r=2018KiB/s,w=2202KiB/s][r=504,w=550 IOPS][eta 00m:29s]
Jobs: 1 (f=1): [m(1)][61.7%][r=1916KiB/s,w=2028KiB/s][r=479,w=507 IOPS][eta 00m:23s]
Jobs: 1 (f=1): [m(1)][71.7%][r=2348KiB/s,w=2016KiB/s][r=587,w=504 IOPS][eta 00m:17s]
Jobs: 1 (f=1): [m(1)][81.7%][r=2170KiB/s,w=2070KiB/s][r=542,w=517 IOPS][eta 00m:11s]
Jobs: 1 (f=1): [m(1)][91.7%][r=2098KiB/s,w=2230KiB/s][r=524,w=557 IOPS][eta 00m:05s]
Jobs: 1 (f=1): [m(1)][100.0%][r=2038KiB/s,w=2126KiB/s][r=509,w=531 IOPS][eta 00m:00s]
TEST: (groupid=0, jobs=1): err= 0: pid=1198248: Wed Feb 21 11:16:41 2024
read: IOPS=527, BW=2110KiB/s (2160kB/s)(124MiB/60001msec)
slat (usec): min=9, max=464, avg=19.47, stdev= 6.89
clat (usec): min=3, max=16846, avg=180.23, stdev=132.30
lat (usec): min=141, max=16866, avg=199.70, stdev=132.51
clat percentiles (usec):
| 1.00th=[ 165], 5.00th=[ 172], 10.00th=[ 174], 20.00th=[ 176],
| 30.00th=[ 178], 40.00th=[ 178], 50.00th=[ 180], 60.00th=[ 180],
| 70.00th=[ 182], 80.00th=[ 182], 90.00th=[ 184], 95.00th=[ 186],
| 99.00th=[ 198], 99.50th=[ 212], 99.90th=[ 383], 99.95th=[ 519],
| 99.99th=[ 922]
bw ( KiB/s): min= 1808, max= 2472, per=100.00%, avg=2112.00, stdev=136.22, samples=119
iops : min= 452, max= 618, avg=528.00, stdev=34.06, samples=119
write: IOPS=528, BW=2113KiB/s (2164kB/s)(124MiB/60001msec); 0 zone resets
slat (usec): min=10, max=695, avg=21.82, stdev=10.01
clat (nsec): min=1648, max=7468.6k, avg=102478.22, stdev=52280.64
lat (usec): min=80, max=7494, avg=124.30, stdev=55.32
clat percentiles (usec):
| 1.00th=[ 70], 5.00th=[ 71], 10.00th=[ 72], 20.00th=[ 73],
| 30.00th=[ 74], 40.00th=[ 114], 50.00th=[ 118], 60.00th=[ 119],
| 70.00th=[ 121], 80.00th=[ 122], 90.00th=[ 123], 95.00th=[ 125],
| 99.00th=[ 135], 99.50th=[ 147], 99.90th=[ 208], 99.95th=[ 314],
| 99.99th=[ 685]
bw ( KiB/s): min= 1920, max= 2296, per=100.00%, avg=2114.15, stdev=73.79, samples=119
iops : min= 480, max= 574, avg=528.54, stdev=18.45, samples=119
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.04%
lat (usec) : 100=19.21%, 250=80.59%, 500=0.10%, 750=0.03%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
fsync/fdatasync/sync_file_range:
sync (usec): min=197, max=26452, avg=925.15, stdev=760.17
sync percentiles (usec):
| 1.00th=[ 338], 5.00th=[ 347], 10.00th=[ 351], 20.00th=[ 355],
| 30.00th=[ 355], 40.00th=[ 359], 50.00th=[ 416], 60.00th=[ 474],
| 70.00th=[ 1467], 80.00th=[ 2024], 90.00th=[ 2089], 95.00th=[ 2089],
| 99.00th=[ 2147], 99.50th=[ 2180], 99.90th=[ 3064], 99.95th=[ 3064],
| 99.99th=[ 8356]
cpu : usr=0.61%, sys=2.87%, ctx=182526, majf=0, minf=9
IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=31647,31700,0,63343 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=2110KiB/s (2160kB/s), 2110KiB/s-2110KiB/s (2160kB/s-2160kB/s), io=124MiB (130MB), run=60001-60001msec
WRITE: bw=2113KiB/s (2164kB/s), 2113KiB/s-2113KiB/s (2164kB/s-2164kB/s), io=124MiB (130MB), run=60001-60001msec
Disk stats (read/write):
sda: ios=31601/109919, merge=27/14981, ticks=6110/52656, in_queue=104054, util=99.89%
```
</details>
#### 1024k read only test
```
$ fio --name TEST --eta-newline=5s --filename=temp.file \
--rw=read --size=500m --io_size=10g --blocksize=1024k \
--ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 \
--numjobs=1 --runtime=60 --group_reporting
[...]
read: IOPS=375, BW=375MiB/s (393MB/s)(10.0GiB/27287msec)
slat (usec): min=6, max=19892, avg=2500.21, stdev=3198.07
clat (msec): min=8, max=160, avg=82.50, stdev=15.20
lat (msec): min=8, max=163, avg=85.00, stdev=15.49
```
<details>
<summary>full output</summary>
```
$ fio --name TEST --eta-newline=5s --filename=temp.file \
--rw=read --size=500m --io_size=10g --blocksize=1024k \
--ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 \
--numjobs=1 --runtime=60 --group_reporting
TEST: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [R(1)][25.9%][r=375MiB/s][r=375 IOPS][eta 00m:20s]
Jobs: 1 (f=1): [R(1)][48.1%][r=374MiB/s][r=374 IOPS][eta 00m:14s]
Jobs: 1 (f=1): [R(1)][70.4%][r=377MiB/s][r=377 IOPS][eta 00m:08s]
Jobs: 1 (f=1): [R(1)][92.6%][r=376MiB/s][r=376 IOPS][eta 00m:02s]
Jobs: 1 (f=1): [R(1)][100.0%][r=375MiB/s][r=375 IOPS][eta 00m:00s]
TEST: (groupid=0, jobs=1): err= 0: pid=1202561: Wed Feb 21 13:13:55 2024
read: IOPS=375, BW=375MiB/s (393MB/s)(10.0GiB/27287msec)
slat (usec): min=6, max=19892, avg=2500.21, stdev=3198.07
clat (msec): min=8, max=160, avg=82.50, stdev=15.20
lat (msec): min=8, max=163, avg=85.00, stdev=15.49
clat percentiles (msec):
| 1.00th=[ 21], 5.00th=[ 73], 10.00th=[ 81], 20.00th=[ 81],
| 30.00th=[ 81], 40.00th=[ 81], 50.00th=[ 81], 60.00th=[ 87],
| 70.00th=[ 87], 80.00th=[ 87], 90.00th=[ 87], 95.00th=[ 93],
| 99.00th=[ 146], 99.50th=[ 153], 99.90th=[ 161], 99.95th=[ 161],
| 99.99th=[ 161]
bw ( KiB/s): min=374784, max=393216, per=100.00%, avg=384417.19, stdev=2687.39, samples=54
iops : min= 366, max= 384, avg=375.41, stdev= 2.62, samples=54
lat (msec) : 10=0.20%, 20=0.64%, 50=2.46%, 100=92.27%, 250=4.43%
cpu : usr=0.15%, sys=1.00%, ctx=5772, majf=0, minf=2056
IO depths : 1=0.2%, 2=0.4%, 4=0.8%, 8=1.6%, 16=3.3%, 32=93.6%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.0%, 32=0.2%, 64=0.0%, >=64=0.0%
issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=375MiB/s (393MB/s), 375MiB/s-375MiB/s (393MB/s-393MB/s), io=10.0GiB (10.7GB), run=27287-27287msec
Disk stats (read/write):
sda: ios=20373/8, merge=0/16, ticks=1520895/4391, in_queue=1525409, util=99.69%
```
</details>
Sign in with Wallet
Connect another wallet