---
# System prepended metadata

title: I/O performance on CCI2

---


# CloudFerro

## from dev-explorer pod

#### 4k randrw test test

```
$ fio --name TEST --eta-newline=5s --filename=temp.file --rw=randrw \
--size=500m --io_size=10g --blocksize=4k --ioengine=libaio --fsync=1 \
--iodepth=1 --direct=1 --numjobs=1 --runtime=60 --group_reporting
[...]
  read: IOPS=38, BW=154KiB/s (158kB/s)(9268KiB/60013msec)
    slat (usec): min=107, max=4817, avg=1064.87, stdev=268.42
    clat (nsec): min=1427, max=56401, avg=3626.08, stdev=2177.33
     lat (usec): min=109, max=4821, avg=1069.57, stdev=268.69
  write: IOPS=40, BW=162KiB/s (166kB/s)(9728KiB/60013msec); 0 zone resets
    slat (usec): min=119, max=4444, avg=247.91, stdev=209.53
    clat (nsec): min=1403, max=59360, avg=2789.82, stdev=2348.27
     lat (usec): min=121, max=4449, avg=251.49, stdev=209.88
```
<details>
<summary>full output</summary>

```
# fio --name TEST --eta-newline=5s --filename=temp.file --rw=randrw \
--size=500m --io_size=10g --blocksize=4k --ioengine=libaio --fsync=1 \
--iodepth=1 --direct=1 --numjobs=1 --runtime=60 --group_reporting

TEST: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.25
Starting 1 process
TEST: Laying out IO file (1 file / 500MiB)
Jobs: 1 (f=1): [m(1)][13.1%][r=164KiB/s,w=164KiB/s][r=41,w=41 IOPS][eta 00m:53s]
Jobs: 1 (f=1): [m(1)][23.0%][r=184KiB/s,w=156KiB/s][r=46,w=39 IOPS][eta 00m:47s]
Jobs: 1 (f=1): [m(1)][32.8%][r=188KiB/s,w=156KiB/s][r=47,w=39 IOPS][eta 00m:41s]
Jobs: 1 (f=1): [m(1)][42.6%][r=148KiB/s,w=164KiB/s][r=37,w=41 IOPS][eta 00m:35s]
Jobs: 1 (f=1): [m(1)][50.8%][r=184KiB/s,w=156KiB/s][r=46,w=39 IOPS][eta 00m:30s]
Jobs: 1 (f=1): [m(1)][60.7%][r=168KiB/s,w=168KiB/s][r=42,w=42 IOPS][eta 00m:24s]
Jobs: 1 (f=1): [m(1)][70.5%][r=151KiB/s,w=171KiB/s][r=37,w=42 IOPS][eta 00m:18s]
Jobs: 1 (f=1): [m(1)][80.3%][r=144KiB/s,w=152KiB/s][r=36,w=38 IOPS][eta 00m:12s]
Jobs: 1 (f=1): [m(1)][90.2%][r=176KiB/s,w=168KiB/s][r=44,w=42 IOPS][eta 00m:06s]
Jobs: 1 (f=1): [m(1)][100.0%][r=140KiB/s,w=152KiB/s][r=35,w=38 IOPS][eta 00m:00s]
TEST: (groupid=0, jobs=1): err= 0: pid=1553: Wed Feb 21 10:14:48 2024
  read: IOPS=38, BW=154KiB/s (158kB/s)(9268KiB/60013msec)
    slat (usec): min=107, max=4817, avg=1064.87, stdev=268.42
    clat (nsec): min=1427, max=56401, avg=3626.08, stdev=2177.33
     lat (usec): min=109, max=4821, avg=1069.57, stdev=268.69
    clat percentiles (nsec):
     |  1.00th=[ 2064],  5.00th=[ 2864], 10.00th=[ 2992], 20.00th=[ 3152],
     | 30.00th=[ 3216], 40.00th=[ 3248], 50.00th=[ 3312], 60.00th=[ 3344],
     | 70.00th=[ 3376], 80.00th=[ 3440], 90.00th=[ 3856], 95.00th=[ 6048],
     | 99.00th=[18048], 99.50th=[19840], 99.90th=[24704], 99.95th=[28800],
     | 99.99th=[56576]
   bw (  KiB/s): min=   64, max=  256, per=99.72%, avg=154.68, stdev=39.27, samples=119
   iops        : min=   16, max=   64, avg=38.66, stdev= 9.82, samples=119
  write: IOPS=40, BW=162KiB/s (166kB/s)(9728KiB/60013msec); 0 zone resets
    slat (usec): min=119, max=4444, avg=247.91, stdev=209.53
    clat (nsec): min=1403, max=59360, avg=2789.82, stdev=2348.27
     lat (usec): min=121, max=4449, avg=251.49, stdev=209.88
    clat percentiles (nsec):
     |  1.00th=[ 1544],  5.00th=[ 1656], 10.00th=[ 1752], 20.00th=[ 2128],
     | 30.00th=[ 2288], 40.00th=[ 2384], 50.00th=[ 2448], 60.00th=[ 2512],
     | 70.00th=[ 2640], 80.00th=[ 3088], 90.00th=[ 3408], 95.00th=[ 3856],
     | 99.00th=[17024], 99.50th=[19328], 99.90th=[32128], 99.95th=[32128],
     | 99.99th=[59136]
   bw (  KiB/s): min=  128, max=  208, per=99.94%, avg=162.41, stdev=14.94, samples=119
   iops        : min=   32, max=   52, avg=40.60, stdev= 3.74, samples=119
  lat (usec)   : 2=9.18%, 4=84.40%, 10=5.14%, 20=0.91%, 50=0.34%
  lat (usec)   : 100=0.04%
  fsync/fdatasync/sync_file_range:
    sync (nsec): min=120, max=16557, avg=553.60, stdev=670.96
    sync percentiles (nsec):
     |  1.00th=[  197],  5.00th=[  245], 10.00th=[  270], 20.00th=[  362],
     | 30.00th=[  506], 40.00th=[  524], 50.00th=[  532], 60.00th=[  540],
     | 70.00th=[  548], 80.00th=[  572], 90.00th=[  660], 95.00th=[  956],
     | 99.00th=[ 1400], 99.50th=[ 1768], 99.90th=[15808], 99.95th=[16320],
     | 99.99th=[16512]
  cpu          : usr=0.20%, sys=0.46%, ctx=11563, majf=0, minf=15
  IO depths    : 1=199.9%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2317,2432,0,4746 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=154KiB/s (158kB/s), 154KiB/s-154KiB/s (158kB/s-158kB/s), io=9268KiB (9490kB), run=60013-60013msec
  WRITE: bw=162KiB/s (166kB/s), 162KiB/s-162KiB/s (166kB/s-166kB/s), io=9728KiB (9961kB), run=60013-60013msec
```
</details>

#### 1024k read only test

```
fio --name TEST --eta-newline=5s --filename=temp.file \
--rw=read --size=500m --io_size=10g --blocksize=1024k \
--ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 \
--numjobs=1 --runtime=60 --group_reporting
[...]
  read: IOPS=467, BW=467MiB/s (490MB/s)(10.0GiB/21915msec)
    slat (usec): min=1389, max=20985, avg=2132.16, stdev=742.66
    clat (usec): min=5, max=187434, avg=64166.10, stdev=21443.59
     lat (usec): min=1710, max=194251, avg=66299.37, stdev=22011.08
```
<details>
<summary>full output</summary>

```
fio --name TEST --eta-newline=5s --filename=temp.file --rw=read --size=500m --io_size=10g --blocksize=1024k --ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=60 --group_reporting
TEST: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
fio-3.25
Starting 1 process
Jobs: 1 (f=1): [R(1)][25.9%][r=493MiB/s][r=493 IOPS][eta 00m:20s]
Jobs: 1 (f=1): [R(1)][56.5%][r=503MiB/s][r=502 IOPS][eta 00m:10s]
Jobs: 1 (f=1): [R(1)][86.4%][r=498MiB/s][r=498 IOPS][eta 00m:03s]
Jobs: 1 (f=1): [R(1)][100.0%][r=516MiB/s][r=515 IOPS][eta 00m:00s]
TEST: (groupid=0, jobs=1): err= 0: pid=1556: Wed Feb 21 11:56:41 2024
  read: IOPS=467, BW=467MiB/s (490MB/s)(10.0GiB/21915msec)
    slat (usec): min=1389, max=20985, avg=2132.16, stdev=742.66
    clat (usec): min=5, max=187434, avg=64166.10, stdev=21443.59
     lat (usec): min=1710, max=194251, avg=66299.37, stdev=22011.08
    clat percentiles (msec):
     |  1.00th=[   10],  5.00th=[   48], 10.00th=[   59], 20.00th=[   60],
     | 30.00th=[   61], 40.00th=[   62], 50.00th=[   62], 60.00th=[   63],
     | 70.00th=[   64], 80.00th=[   65], 90.00th=[   67], 95.00th=[   71],
     | 99.00th=[  155], 99.50th=[  157], 99.90th=[  182], 99.95th=[  182],
     | 99.99th=[  188]
   bw (  KiB/s): min=124928, max=575488, per=99.60%, avg=476540.49, stdev=103255.37, samples=43
   iops        : min=  122, max=  562, avg=465.35, stdev=100.82, samples=43
  lat (usec)   : 10=0.19%, 20=0.01%, 50=0.01%
  lat (msec)   : 2=0.11%, 4=0.21%, 10=0.55%, 20=1.00%, 50=3.11%
  lat (msec)   : 100=90.16%, 250=4.67%
  cpu          : usr=0.38%, sys=5.48%, ctx=10566, majf=0, minf=8205
  IO depths    : 1=0.2%, 2=0.4%, 4=0.8%, 8=1.6%, 16=3.3%, 32=93.6%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.0%, 32=0.2%, 64=0.0%, >=64=0.0%
     issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=467MiB/s (490MB/s), 467MiB/s-467MiB/s (490MB/s-490MB/s), io=10.0GiB (10.7GB), run=21915-21915msec
```
</details>

## from database-cluster pod

#### 4k randrw test test

```
$ fio --name TEST --eta-newline=5s --filename=temp.file --rw=randrw \
--size=500m --io_size=10g --blocksize=4k --ioengine=libaio --fsync=1 \
--iodepth=1 --direct=1 --numjobs=1 --runtime=60 --group_reporting
[...]
   read: IOPS=37, BW=148KiB/s (152kB/s)(8896KiB/60021msec)
    slat (usec): min=115, max=5894, avg=1005.91, stdev=234.89
    clat (nsec): min=1696, max=159360, avg=3387.23, stdev=3649.95
     lat (usec): min=117, max=5899, avg=1011.13, stdev=236.61
  write: IOPS=38, BW=156KiB/s (160kB/s)(9352KiB/60021msec)
    slat (usec): min=121, max=1593, avg=203.09, stdev=74.26
    clat (nsec): min=1561, max=155816, avg=2956.79, stdev=3536.41
     lat (usec): min=124, max=1598, avg=207.03, stdev=75.27
```
<details>
<summary>full output</summary>

```
TEST: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.1
Starting 1 process
TEST: Laying out IO file (1 file / 500MiB)
Jobs: 1 (f=1): [m(1)][13.1%][r=100KiB/s,w=148KiB/s][r=25,w=37 IOPS][eta 00m:53s]
Jobs: 1 (f=1): [m(1)][23.0%][r=148KiB/s,w=168KiB/s][r=37,w=42 IOPS][eta 00m:47s]
Jobs: 1 (f=1): [m(1)][32.8%][r=156KiB/s,w=168KiB/s][r=39,w=42 IOPS][eta 00m:41s]
Jobs: 1 (f=1): [m(1)][42.6%][r=156KiB/s,w=168KiB/s][r=39,w=42 IOPS][eta 00m:35s]
Jobs: 1 (f=1): [m(1)][52.5%][r=172KiB/s,w=152KiB/s][r=43,w=38 IOPS][eta 00m:29s]
Jobs: 1 (f=1): [m(1)][62.3%][r=172KiB/s,w=168KiB/s][r=43,w=42 IOPS][eta 00m:23s]
Jobs: 1 (f=1): [m(1)][72.1%][r=196KiB/s,w=148KiB/s][r=49,w=37 IOPS][eta 00m:17s]
Jobs: 1 (f=1): [m(1)][82.0%][r=200KiB/s,w=156KiB/s][r=50,w=39 IOPS][eta 00m:11s]
Jobs: 1 (f=1): [m(1)][91.8%][r=168KiB/s,w=136KiB/s][r=42,w=34 IOPS][eta 00m:05s]
Jobs: 1 (f=1): [m(1)][100.0%][r=96KiB/s,w=136KiB/s][r=24,w=34 IOPS][eta 00m:00s]
TEST: (groupid=0, jobs=1): err= 0: pid=1558569: Thu Feb 22 13:54:40 2024
read: IOPS=37, BW=148KiB/s (152kB/s)(8896KiB/60021msec)
slat (usec): min=115, max=5894, avg=1005.91, stdev=234.89
clat (nsec): min=1696, max=159360, avg=3387.23, stdev=3649.95
 lat (usec): min=117, max=5899, avg=1011.13, stdev=236.61
clat percentiles (usec):
 |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
 | 30.00th=[    3], 40.00th=[    3], 50.00th=[    4], 60.00th=[    4],
 | 70.00th=[    4], 80.00th=[    4], 90.00th=[    4], 95.00th=[    5],
 | 99.00th=[    8], 99.50th=[   19], 99.90th=[   24], 99.95th=[   31],
 | 99.99th=[  159]
bw (  KiB/s): min=   64, max=  264, per=100.00%, avg=148.10, stdev=45.51, samples=120
iops        : min=   16, max=   66, avg=36.90, stdev=11.38, samples=120
write: IOPS=38, BW=156KiB/s (160kB/s)(9352KiB/60021msec)
slat (usec): min=121, max=1593, avg=203.09, stdev=74.26
clat (nsec): min=1561, max=155816, avg=2956.79, stdev=3536.41
 lat (usec): min=124, max=1598, avg=207.03, stdev=75.27
clat percentiles (nsec):
 |  1.00th=[  1832],  5.00th=[  1976], 10.00th=[  2128], 20.00th=[  2416],
 | 30.00th=[  2512], 40.00th=[  2608], 50.00th=[  2704], 60.00th=[  2832],
 | 70.00th=[  2960], 80.00th=[  3152], 90.00th=[  3440], 95.00th=[  3696],
 | 99.00th=[  5152], 99.50th=[ 17280], 99.90th=[ 20096], 99.95th=[ 42240],
 | 99.99th=[156672]
bw (  KiB/s): min=  104, max=  200, per=100.00%, avg=155.70, stdev=19.43, samples=120
iops        : min=   26, max=   50, avg=38.80, stdev= 4.91, samples=120
lat (usec)   : 2=2.92%, 4=92.88%, 10=3.33%, 20=0.70%, 50=0.13%
lat (usec)   : 250=0.04%
cpu          : usr=0.18%, sys=0.50%, ctx=10481, majf=0, minf=8
IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
 submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwt: total=2224,2338,0, short=0,0,0, dropped=0,0,0
 latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
READ: bw=148KiB/s (152kB/s), 148KiB/s-148KiB/s (152kB/s-152kB/s), io=8896KiB (9110kB), run=60021-60021msec
WRITE: bw=156KiB/s (160kB/s), 156KiB/s-156KiB/s (160kB/s-160kB/s), io=9352KiB (9576kB), run=60021-60021msec
```

</details>


# CCI2

## from master node

#### 4k randrw test

```
$ fio --name TEST --eta-newline=5s --filename=temp.file --rw=randrw \
--size=500m --io_size=10g --blocksize=4k --ioengine=libaio --fsync=1 \
--iodepth=1 --direct=1 --numjobs=1 --runtime=60 --group_reporting
[...]
  read: IOPS=37, BW=149KiB/s (153kB/s)(8948KiB/60020msec)
    slat (usec): min=11, max=235, avg=35.57, stdev=16.18
    clat (usec): min=50, max=6479, avg=933.29, stdev=284.90
     lat (usec): min=63, max=6535, avg=969.63, stdev=286.48
  write: IOPS=39, BW=157KiB/s (161kB/s)(9416KiB/60020msec); 0 zone resets
    slat (usec): min=19, max=779, avg=40.63, stdev=21.09
    clat (usec): min=66, max=4534, avg=136.89, stdev=147.64
     lat (usec): min=99, max=4573, avg=178.32, stdev=149.73
```

<details>
<summary>full output</summary>

    
```
fio --name TEST --eta-newline=5s --filename=temp.file --rw=randrw \
--size=500m --io_size=10g --blocksize=4k --ioengine=libaio --fsync=1 \
--iodepth=1 --direct=1 --numjobs=1 --runtime=60 --group_reporting

TEST: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.16
Starting 1 process
TEST: Laying out IO file (1 file / 400MiB)
Jobs: 1 (f=1): [m(1)][13.1%][r=184KiB/s,w=164KiB/s][r=46,w=41 IOPS][eta 00m:53s]
Jobs: 1 (f=1): [m(1)][23.0%][r=196KiB/s,w=160KiB/s][r=49,w=40 IOPS][eta 00m:47s]
Jobs: 1 (f=1): [m(1)][32.8%][r=184KiB/s,w=148KiB/s][r=46,w=37 IOPS][eta 00m:41s]
Jobs: 1 (f=1): [m(1)][42.6%][r=132KiB/s,w=164KiB/s][r=33,w=41 IOPS][eta 00m:35s]
Jobs: 1 (f=1): [m(1)][52.5%][r=188KiB/s,w=160KiB/s][r=47,w=40 IOPS][eta 00m:29s]
Jobs: 1 (f=1): [m(1)][62.3%][r=160KiB/s,w=152KiB/s][r=40,w=38 IOPS][eta 00m:23s]
Jobs: 1 (f=1): [m(1)][72.1%][r=140KiB/s,w=164KiB/s][r=35,w=41 IOPS][eta 00m:17s]
Jobs: 1 (f=1): [m(1)][82.0%][r=156KiB/s,w=156KiB/s][r=39,w=39 IOPS][eta 00m:11s]
Jobs: 1 (f=1): [m(1)][91.8%][r=164KiB/s,w=128KiB/s][r=41,w=32 IOPS][eta 00m:05s]
Jobs: 1 (f=1): [m(1)][100.0%][r=112KiB/s,w=168KiB/s][r=28,w=42 IOPS][eta 00m:00s]
TEST: (groupid=0, jobs=1): err= 0: pid=979671: Wed Feb 21 10:07:52 2024
  read: IOPS=37, BW=149KiB/s (153kB/s)(8948KiB/60020msec)
    slat (usec): min=11, max=235, avg=35.57, stdev=16.18
    clat (usec): min=50, max=6479, avg=933.29, stdev=284.90
     lat (usec): min=63, max=6535, avg=969.63, stdev=286.48
    clat percentiles (usec):
     |  1.00th=[   77],  5.00th=[  660], 10.00th=[  783], 20.00th=[  848],
     | 30.00th=[  889], 40.00th=[  922], 50.00th=[  947], 60.00th=[  979],
     | 70.00th=[ 1012], 80.00th=[ 1057], 90.00th=[ 1123], 95.00th=[ 1188],
     | 99.00th=[ 1336], 99.50th=[ 1483], 99.90th=[ 4228], 99.95th=[ 5014],
     | 99.99th=[ 6456]
   bw (  KiB/s): min=   64, max=  248, per=100.00%, avg=149.11, stdev=43.54, samples=120
   iops        : min=   16, max=   62, avg=37.26, stdev=10.89, samples=120
  write: IOPS=39, BW=157KiB/s (161kB/s)(9416KiB/60020msec); 0 zone resets
    slat (usec): min=19, max=779, avg=40.63, stdev=21.09
    clat (usec): min=66, max=4534, avg=136.89, stdev=147.64
     lat (usec): min=99, max=4573, avg=178.32, stdev=149.73
    clat percentiles (usec):
     |  1.00th=[   83],  5.00th=[   91], 10.00th=[   96], 20.00th=[  102],
     | 30.00th=[  111], 40.00th=[  118], 50.00th=[  124], 60.00th=[  129],
     | 70.00th=[  137], 80.00th=[  147], 90.00th=[  163], 95.00th=[  184],
     | 99.00th=[  412], 99.50th=[  635], 99.90th=[ 2802], 99.95th=[ 3359],
     | 99.99th=[ 4555]
   bw (  KiB/s): min=  112, max=  184, per=100.00%, avg=156.91, stdev=17.38, samples=120
   iops        : min=   28, max=   46, avg=39.21, stdev= 4.34, samples=120
  lat (usec)   : 100=10.63%, 250=42.13%, 500=0.46%, 750=1.46%, 1000=28.64%
  lat (msec)   : 2=16.51%, 4=0.09%, 10=0.09%
  fsync/fdatasync/sync_file_range:
    sync (nsec): min=143, max=70365, avg=794.85, stdev=1465.69
    sync percentiles (nsec):
     |  1.00th=[  197],  5.00th=[  330], 10.00th=[  362], 20.00th=[  414],
     | 30.00th=[  482], 40.00th=[  506], 50.00th=[  564], 60.00th=[  700],
     | 70.00th=[  900], 80.00th=[ 1096], 90.00th=[ 1256], 95.00th=[ 1432],
     | 99.00th=[ 1864], 99.50th=[ 4448], 99.90th=[16320], 99.95th=[16512],
     | 99.99th=[70144]
  cpu          : usr=0.14%, sys=0.44%, ctx=13789, majf=0, minf=16
  IO depths    : 1=199.9%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2237,2354,0,4588 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=149KiB/s (153kB/s), 149KiB/s-149KiB/s (153kB/s-153kB/s), io=8948KiB (9163kB), run=60020-60020msec
  WRITE: bw=157KiB/s (161kB/s), 157KiB/s-157KiB/s (161kB/s-161kB/s), io=9416KiB (9642kB), run=60020-60020msec

Disk stats (read/write):
  vda: ios=2236/10421, merge=0/3501, ticks=2089/57269, in_queue=47824, util=99.88%
```
</details>

#### 1024k read only test

```
$ fio --name TEST --eta-newline=5s --filename=temp.file \
--rw=read --size=500m --io_size=10g --blocksize=1024k \
--ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 \
--numjobs=1 --runtime=60 --group_reporting
[...]
  read: IOPS=2185, BW=2185MiB/s (2291MB/s)(10.0GiB/4686msec)
    slat (usec): min=24, max=1789, avg=53.99, stdev=80.70
    clat (usec): min=879, max=251152, avg=14385.95, stdev=19743.81
     lat (usec): min=937, max=251248, avg=14440.49, stdev=19744.72
```

<details>
<summary>full output</summary>
    
```
$ fio --name TEST --eta-newline=5s --filename=temp.file --rw=read --size=500m --io_size=10g --blocksize=1024k --ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=60 --group_reporting
TEST: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
fio-3.16
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=2299MiB/s][r=2299 IOPS][eta 00m:00s]
TEST: (groupid=0, jobs=1): err= 0: pid=980507: Wed Feb 21 12:05:54 2024
  read: IOPS=2185, BW=2185MiB/s (2291MB/s)(10.0GiB/4686msec)
    slat (usec): min=24, max=1789, avg=53.99, stdev=80.70
    clat (usec): min=879, max=251152, avg=14385.95, stdev=19743.81
     lat (usec): min=937, max=251248, avg=14440.49, stdev=19744.72
    clat percentiles (msec):
     |  1.00th=[    3],  5.00th=[    5], 10.00th=[    7], 20.00th=[    9],
     | 30.00th=[   10], 40.00th=[   11], 50.00th=[   12], 60.00th=[   13],
     | 70.00th=[   14], 80.00th=[   15], 90.00th=[   18], 95.00th=[   23],
     | 99.00th=[  146], 99.50th=[  192], 99.90th=[  230], 99.95th=[  251],
     | 99.99th=[  251]
   bw (  MiB/s): min= 1774, max= 2668, per=99.48%, avg=2173.78, stdev=296.21, samples=9
   iops        : min= 1774, max= 2668, avg=2173.78, stdev=296.21, samples=9
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=0.14%, 4=2.70%, 10=29.80%, 20=60.35%, 50=5.17%
  lat (msec)   : 100=0.74%, 250=1.08%, 500=0.01%
  cpu          : usr=2.09%, sys=13.28%, ctx=9541, majf=0, minf=8204
  IO depths    : 1=0.2%, 2=0.4%, 4=0.8%, 8=1.6%, 16=3.3%, 32=93.6%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.0%, 32=0.2%, 64=0.0%, >=64=0.0%
     issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=2185MiB/s (2291MB/s), 2185MiB/s-2185MiB/s (2291MB/s-2291MB/s), io=10.0GiB (10.7GB), run=4686-4686msec

Disk stats (read/write):
  vda: ios=10166/35, merge=0/1, ticks=120695/202, in_queue=101072, util=98.04%
```
</details>

## from database-cluster

#### 4k randrw test

```
$ fio --name TEST --eta-newline=5s --filename=temp.file --rw=randrw \
--size=500m --io_size=10g --blocksize=4k --ioengine=libaio --fsync=1 \
--iodepth=1 --direct=1 --numjobs=1 --runtime=60 --group_reporting
[...]
   read: IOPS=13, BW=54.5KiB/s (55.8kB/s)(3272KiB/60042msec)
    slat (usec): min=10, max=318, avg=30.90, stdev=22.03
    clat (usec): min=332, max=447433, avg=17217.89, stdev=28268.72
     lat (usec): min=344, max=447451, avg=17249.26, stdev=28268.38
  write: IOPS=14, BW=57.8KiB/s (59.1kB/s)(3468KiB/60042msec)
    slat (usec): min=10, max=229, avg=34.06, stdev=24.61
    clat (usec): min=42, max=834, avg=101.42, stdev=49.01
     lat (usec): min=65, max=897, avg=135.94, stdev=60.27
```

<details>
<summary>full output</summary>

```
# fio --name TEST --eta-newline=5s --filename=temp.file --rw=randrw \
> --size=500m --io_size=10g --blocksize=4k --ioengine=libaio --fsync=1 \
> --iodepth=1 --direct=1 --numjobs=1 --runtime=60 --group_reporting
TEST: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.1
Starting 1 process
TEST: Laying out IO file (1 file / 500MiB)
Jobs: 1 (f=1): [m(1)][13.1%][r=84KiB/s,w=132KiB/s][r=21,w=33 IOPS][eta 00m:53s]
Jobs: 1 (f=1): [m(1)][23.0%][r=132KiB/s,w=144KiB/s][r=33,w=36 IOPS][eta 00m:47s]
Jobs: 1 (f=1): [m(1)][32.8%][r=124KiB/s,w=108KiB/s][r=31,w=27 IOPS][eta 00m:41s]
Jobs: 1 (f=1): [m(1)][42.6%][r=108KiB/s,w=152KiB/s][r=27,w=38 IOPS][eta 00m:35s]
Jobs: 1 (f=1): [m(1)][52.5%][r=4KiB/s,w=16KiB/s][r=1,w=4 IOPS][eta 00m:29s]
Jobs: 1 (f=1): [m(1)][62.3%][r=32KiB/s,w=28KiB/s][r=8,w=7 IOPS][eta 00m:23s]
Jobs: 1 (f=1): [m(1)][72.1%][r=44KiB/s,w=16KiB/s][r=11,w=4 IOPS][eta 00m:17s]
Jobs: 1 (f=1): [m(1)][82.0%][r=16KiB/s,w=20KiB/s][r=4,w=5 IOPS][eta 00m:11s]
Jobs: 1 (f=1): [m(1)][91.8%][r=72KiB/s,w=56KiB/s][r=18,w=14 IOPS][eta 00m:05s]
Jobs: 1 (f=1): [m(1)][100.0%][r=40KiB/s,w=60KiB/s][r=10,w=15 IOPS][eta 00m:00s]
TEST: (groupid=0, jobs=1): err= 0: pid=600258: Thu Feb 22 13:43:52 2024
   read: IOPS=13, BW=54.5KiB/s (55.8kB/s)(3272KiB/60042msec)
    slat (usec): min=10, max=318, avg=30.90, stdev=22.03
    clat (usec): min=332, max=447433, avg=17217.89, stdev=28268.72
     lat (usec): min=344, max=447451, avg=17249.26, stdev=28268.38
    clat percentiles (usec):
     |  1.00th=[   433],  5.00th=[   498], 10.00th=[   545], 20.00th=[   742],
     | 30.00th=[  5407], 40.00th=[  9503], 50.00th=[ 11600], 60.00th=[ 13435],
     | 70.00th=[ 15795], 80.00th=[ 21627], 90.00th=[ 36439], 95.00th=[ 57410],
     | 99.00th=[135267], 99.50th=[162530], 99.90th=[446694], 99.95th=[446694],
     | 99.99th=[446694]
   bw (  KiB/s): min=    7, max=  192, per=100.00%, avg=57.79, stdev=46.70, samples=113
   iops        : min=    1, max=   48, avg=14.41, stdev=11.67, samples=113
  write: IOPS=14, BW=57.8KiB/s (59.1kB/s)(3468KiB/60042msec)
    slat (usec): min=10, max=229, avg=34.06, stdev=24.61
    clat (usec): min=42, max=834, avg=101.42, stdev=49.01
     lat (usec): min=65, max=897, avg=135.94, stdev=60.27
    clat percentiles (usec):
     |  1.00th=[   54],  5.00th=[   62], 10.00th=[   68], 20.00th=[   75],
     | 30.00th=[   82], 40.00th=[   88], 50.00th=[   94], 60.00th=[  101],
     | 70.00th=[  109], 80.00th=[  119], 90.00th=[  141], 95.00th=[  161],
     | 99.00th=[  212], 99.50th=[  310], 99.90th=[  832], 99.95th=[  832],
     | 99.99th=[  832]
   bw (  KiB/s): min=    7, max=  216, per=100.00%, avg=59.22, stdev=54.42, samples=117
   iops        : min=    1, max=   54, avg=14.76, stdev=13.62, samples=117
  lat (usec)   : 50=0.06%, 100=30.33%, 250=20.65%, 500=2.91%, 750=7.36%
  lat (usec)   : 1000=2.31%
  lat (msec)   : 2=1.13%, 4=0.36%, 10=6.94%, 20=17.15%, 50=7.54%
  lat (msec)   : 100=2.26%, 250=0.95%, 500=0.06%
  cpu          : usr=0.04%, sys=0.14%, ctx=4275, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=818,867,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=54.5KiB/s (55.8kB/s), 54.5KiB/s-54.5KiB/s (55.8kB/s-55.8kB/s), io=3272KiB (3351kB), run=60042-60042msec
  WRITE: bw=57.8KiB/s (59.1kB/s), 57.8KiB/s-57.8KiB/s (59.1kB/s-59.1kB/s), io=3468KiB (3551kB), run=60042-60042msec
```

</details>

## from master node - SSD

#### 4k randrw test

```
$ fio --name TEST --eta-newline=5s --filename=temp.file --rw=randrw \
--size=500m --io_size=10g --blocksize=4k --ioengine=libaio --fsync=1 \
--iodepth=1 --direct=1 --numjobs=1 --runtime=60 --group_reporting
[...]
  read: IOPS=567, BW=2270KiB/s (2325kB/s)(133MiB/60001msec)
    slat (usec): min=5, max=2285, avg=14.30, stdev=15.75
    clat (usec): min=2, max=4571, avg=318.91, stdev=93.54
     lat (usec): min=143, max=4582, avg=333.35, stdev=95.06
   write: IOPS=568, BW=2274KiB/s (2329kB/s)(133MiB/60001msec); 0 zone resets
    slat (usec): min=5, max=927, avg=14.76, stdev= 8.28
    clat (nsec): min=1052, max=70555k, avg=66902.94, stdev=383045.02
     lat (usec): min=40, max=70565, avg=81.80, stdev=383.19
```

<details>
<summary>full output</summary>

```
    fio --name TEST --eta-newline=5s --filename=temp.file --rw=randrw --size=2g --io_size=10g --blocksize=4k --ioengine=libaio --fsync=1 --iodepth=1 --direct=1 --numjobs=1 --runtime=60 --group_reporting
TEST: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [m(1)][11.7%][r=2306KiB/s,w=2210KiB/s][r=576,w=552 IOPS][eta 00m:53s]
Jobs: 1 (f=1): [m(1)][21.7%][r=2336KiB/s,w=2288KiB/s][r=584,w=572 IOPS][eta 00m:47s] 
Jobs: 1 (f=1): [m(1)][31.7%][r=2182KiB/s,w=2122KiB/s][r=545,w=530 IOPS][eta 00m:41s] 
Jobs: 1 (f=1): [m(1)][41.7%][r=2272KiB/s,w=2304KiB/s][r=568,w=576 IOPS][eta 00m:35s] 
Jobs: 1 (f=1): [m(1)][51.7%][r=2268KiB/s,w=2360KiB/s][r=567,w=590 IOPS][eta 00m:29s] 
Jobs: 1 (f=1): [m(1)][61.7%][r=2362KiB/s,w=2270KiB/s][r=590,w=567 IOPS][eta 00m:23s] 
Jobs: 1 (f=1): [m(1)][71.7%][r=2148KiB/s,w=2284KiB/s][r=537,w=571 IOPS][eta 00m:17s] 
Jobs: 1 (f=1): [m(1)][81.7%][r=2064KiB/s,w=2364KiB/s][r=516,w=591 IOPS][eta 00m:11s] 
Jobs: 1 (f=1): [m(1)][91.7%][r=2284KiB/s,w=2288KiB/s][r=571,w=572 IOPS][eta 00m:05s] 
Jobs: 1 (f=1): [m(1)][100.0%][r=2122KiB/s,w=2190KiB/s][r=530,w=547 IOPS][eta 00m:00s]
TEST: (groupid=0, jobs=1): err= 0: pid=582748: Thu Feb 22 09:48:34 2024
  read: IOPS=567, BW=2270KiB/s (2325kB/s)(133MiB/60001msec)
    slat (usec): min=5, max=2285, avg=14.30, stdev=15.75
    clat (usec): min=2, max=4571, avg=318.91, stdev=93.54
     lat (usec): min=143, max=4582, avg=333.35, stdev=95.06
    clat percentiles (usec):
     |  1.00th=[  174],  5.00th=[  200], 10.00th=[  223], 20.00th=[  273],
     | 30.00th=[  293], 40.00th=[  306], 50.00th=[  318], 60.00th=[  330],
     | 70.00th=[  343], 80.00th=[  359], 90.00th=[  388], 95.00th=[  424],
     | 99.00th=[  570], 99.50th=[  652], 99.90th=[ 1106], 99.95th=[ 1745],
     | 99.99th=[ 3294]
   bw (  KiB/s): min= 1904, max= 2776, per=100.00%, avg=2271.15, stdev=167.61, samples=119
   iops        : min=  476, max=  694, avg=567.78, stdev=41.91, samples=119
  write: IOPS=568, BW=2274KiB/s (2329kB/s)(133MiB/60001msec); 0 zone resets
    slat (usec): min=5, max=927, avg=14.76, stdev= 8.28
    clat (nsec): min=1052, max=70555k, avg=66902.94, stdev=383045.02
     lat (usec): min=40, max=70565, avg=81.80, stdev=383.19
    clat percentiles (usec):
     |  1.00th=[   39],  5.00th=[   45], 10.00th=[   48], 20.00th=[   51],
     | 30.00th=[   55], 40.00th=[   59], 50.00th=[   62], 60.00th=[   65],
     | 70.00th=[   69], 80.00th=[   73], 90.00th=[   82], 95.00th=[   94],
     | 99.00th=[  145], 99.50th=[  186], 99.90th=[  334], 99.95th=[  420],
     | 99.99th=[  832]
   bw (  KiB/s): min= 1920, max= 2688, per=100.00%, avg=2276.46, stdev=143.75, samples=119
   iops        : min=  480, max=  672, avg=569.11, stdev=35.94, samples=119
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=8.59%
  lat (usec)   : 100=39.54%, 250=8.99%, 500=41.80%, 750=0.91%, 1000=0.07%
  lat (msec)   : 2=0.05%, 4=0.02%, 10=0.01%, 100=0.01%
  fsync/fdatasync/sync_file_range:
    sync (nsec): min=50, max=10210k, avg=346.93, stdev=39105.95
    sync percentiles (nsec):
     |  1.00th=[   81],  5.00th=[  100], 10.00th=[  110], 20.00th=[  120],
     | 30.00th=[  131], 40.00th=[  151], 50.00th=[  161], 60.00th=[  181],
     | 70.00th=[  201], 80.00th=[  251], 90.00th=[  310], 95.00th=[  382],
     | 99.00th=[  502], 99.50th=[  580], 99.90th=[ 2192], 99.95th=[ 7264],
     | 99.99th=[14400]
  cpu          : usr=0.81%, sys=2.72%, ctx=176222, majf=0, minf=20
  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=34051,34117,0,68165 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=2270KiB/s (2325kB/s), 2270KiB/s-2270KiB/s (2325kB/s-2325kB/s), io=133MiB (139MB), run=60001-60001msec
  WRITE: bw=2274KiB/s (2329kB/s), 2274KiB/s-2274KiB/s (2329kB/s-2329kB/s), io=133MiB (140MB), run=60001-60001msec

Disk stats (read/write):
  vda: ios=33988/133726, merge=0/29747, ticks=10962/46793, in_queue=97476, util=99.87%
```
    
</details>

#### 1024k read only test

```
$ fio --name TEST --eta-newline=5s --filename=temp.file \
--rw=read --size=500m --io_size=10g --blocksize=1024k \
--ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 \
--numjobs=1 --runtime=60 --group_reporting
[...]
   read: IOPS=3923, BW=3923MiB/s (4114MB/s)(10.0GiB/2610msec)
    slat (usec): min=21, max=576, avg=35.50, stdev=23.08
    clat (usec): min=1038, max=45490, avg=8091.25, stdev=5350.83
     lat (usec): min=1067, max=45541, avg=8127.05, stdev=5350.84
```

<details>
    <summary>full output</summary>
    
```
    fio --name TEST --eta-newline=5s --filename=temp.file --rw=read --size=2g --io_size=10g --blocksize=1024k --ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=60 --group_reporting
TEST: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
fio-3.28
Starting 1 process
TEST: Laying out IO file (1 file / 2048MiB)
Jobs: 1 (f=1): [R(1)][-.-%][r=3945MiB/s][r=3945 IOPS][eta 00m:00s]
TEST: (groupid=0, jobs=1): err= 0: pid=572342: Thu Feb 22 09:27:42 2024
  read: IOPS=3923, BW=3923MiB/s (4114MB/s)(10.0GiB/2610msec)
    slat (usec): min=21, max=576, avg=35.50, stdev=23.08
    clat (usec): min=1038, max=45490, avg=8091.25, stdev=5350.83
     lat (usec): min=1067, max=45541, avg=8127.05, stdev=5350.84
    clat percentiles (usec):
     |  1.00th=[ 1663],  5.00th=[ 2089], 10.00th=[ 2573], 20.00th=[ 3458],
     | 30.00th=[ 4359], 40.00th=[ 5473], 50.00th=[ 6783], 60.00th=[ 8225],
     | 70.00th=[ 9896], 80.00th=[12256], 90.00th=[15533], 95.00th=[18482],
     | 99.00th=[24249], 99.50th=[28181], 99.90th=[34341], 99.95th=[38011],
     | 99.99th=[43254]
   bw (  MiB/s): min= 3678, max= 4258, per=100.00%, avg=3934.80, stdev=211.05, samples=5
   iops        : min= 3678, max= 4258, avg=3934.80, stdev=211.05, samples=5
  lat (msec)   : 2=4.19%, 4=21.63%, 10=44.33%, 20=26.53%, 50=3.32%
  cpu          : usr=1.61%, sys=15.37%, ctx=9285, majf=0, minf=8204
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.4%, 16=0.8%, 32=98.5%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=3923MiB/s (4114MB/s), 3923MiB/s-3923MiB/s (4114MB/s-4114MB/s), io=10.0GiB (10.7GB), run=2610-2610msec

Disk stats (read/write):
  vda: ios=30041/149, merge=0/31, ticks=239873/890, in_queue=241478, util=96.50%
```
    
</details>

# OVH

## from a pod

#### 4k randrw test

```
$ fio --name TEST --eta-newline=5s --filename=temp.file --rw=randrw \
--size=500m --io_size=10g --blocksize=4k --ioengine=libaio --fsync=1 \
--iodepth=1 --direct=1 --numjobs=1 --runtime=60 --group_reporting
[...]
  read: IOPS=791, BW=3167KiB/s (3243kB/s)(186MiB/60001msec)
    slat (usec): min=7, max=890, avg=34.88, stdev=24.16
    clat (usec): min=6, max=13775, avg=219.72, stdev=139.66
     lat (usec): min=109, max=13814, avg=254.60, stdev=146.79
  write: IOPS=786, BW=3148KiB/s (3223kB/s)(184MiB/60001msec); 0 zone resets
    slat (usec): min=7, max=1147, avg=36.34, stdev=24.94
    clat (nsec): min=1979, max=14352k, avg=190479.97, stdev=158744.77
     lat (usec): min=72, max=14368, avg=226.82, stdev=165.37
```

<details>
    
<summary>full output</summary>

```
$ fio --name TEST --eta-newline=5s --filename=temp.file --rw=randrw \
--size=500m --io_size=10g --blocksize=4k --ioengine=libaio --fsync=1 \
--iodepth=1 --direct=1 --numjobs=1 --runtime=60 --group_reporting
TEST: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [m(1)][11.7%][r=3151KiB/s,w=3015KiB/s][r=787,w=753 IOPS][eta 00m:53s]
Jobs: 1 (f=1): [m(1)][21.7%][r=3984KiB/s,w=3860KiB/s][r=996,w=965 IOPS][eta 00m:47s] 
Jobs: 1 (f=1): [m(1)][31.7%][r=2932KiB/s,w=2936KiB/s][r=733,w=734 IOPS][eta 00m:41s] 
Jobs: 1 (f=1): [m(1)][41.7%][r=3448KiB/s,w=3652KiB/s][r=862,w=913 IOPS][eta 00m:35s] 
Jobs: 1 (f=1): [m(1)][51.7%][r=3072KiB/s,w=2784KiB/s][r=768,w=696 IOPS][eta 00m:29s] 
Jobs: 1 (f=1): [m(1)][61.7%][r=3644KiB/s,w=3784KiB/s][r=911,w=946 IOPS][eta 00m:23s]  
Jobs: 1 (f=1): [m(1)][71.7%][r=3032KiB/s,w=3164KiB/s][r=758,w=791 IOPS][eta 00m:17s] 
Jobs: 1 (f=1): [m(1)][81.7%][r=2510KiB/s,w=2526KiB/s][r=627,w=631 IOPS][eta 00m:11s] 
Jobs: 1 (f=1): [m(1)][90.2%][r=4208KiB/s,w=3900KiB/s][r=1052,w=975 IOPS][eta 00m:06s] 
Jobs: 1 (f=1): [m(1)][100.0%][r=3975KiB/s,w=3919KiB/s][r=993,w=979 IOPS][eta 00m:00s] 
TEST: (groupid=0, jobs=1): err= 0: pid=5181: Wed Feb 21 14:00:42 2024
  read: IOPS=791, BW=3167KiB/s (3243kB/s)(186MiB/60001msec)
    slat (usec): min=7, max=890, avg=34.88, stdev=24.16
    clat (usec): min=6, max=13775, avg=219.72, stdev=139.66
     lat (usec): min=109, max=13814, avg=254.60, stdev=146.79
    clat percentiles (usec):
     |  1.00th=[  135],  5.00th=[  149], 10.00th=[  157], 20.00th=[  167],
     | 30.00th=[  180], 40.00th=[  194], 50.00th=[  206], 60.00th=[  219],
     | 70.00th=[  231], 80.00th=[  245], 90.00th=[  269], 95.00th=[  306],
     | 99.00th=[  627], 99.50th=[  742], 99.90th=[ 1106], 99.95th=[ 1319],
     | 99.99th=[ 4490]
   bw (  KiB/s): min= 2208, max= 6280, per=100.00%, avg=3174.24, stdev=905.25, samples=119
   iops        : min=  552, max= 1570, avg=793.55, stdev=226.31, samples=119
  write: IOPS=786, BW=3148KiB/s (3223kB/s)(184MiB/60001msec); 0 zone resets
    slat (usec): min=7, max=1147, avg=36.34, stdev=24.94
    clat (nsec): min=1979, max=14352k, avg=190479.97, stdev=158744.77
     lat (usec): min=72, max=14368, avg=226.82, stdev=165.37
    clat percentiles (usec):
     |  1.00th=[  102],  5.00th=[  116], 10.00th=[  126], 20.00th=[  137],
     | 30.00th=[  149], 40.00th=[  167], 50.00th=[  182], 60.00th=[  194],
     | 70.00th=[  206], 80.00th=[  225], 90.00th=[  251], 95.00th=[  281],
     | 99.00th=[  478], 99.50th=[  627], 99.90th=[ 1020], 99.95th=[ 1401],
     | 99.99th=[ 8979]
   bw (  KiB/s): min= 2136, max= 6048, per=100.00%, avg=3152.46, stdev=864.44, samples=119
   iops        : min=  534, max= 1512, avg=788.11, stdev=216.11, samples=119
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.02%, 20=0.01%, 50=0.01%
  lat (usec)   : 100=0.35%, 250=86.02%, 500=12.05%, 750=1.19%, 1000=0.24%
  lat (msec)   : 2=0.10%, 4=0.01%, 10=0.01%, 20=0.01%
  fsync/fdatasync/sync_file_range:
    sync (usec): min=120, max=15030, avg=594.10, stdev=473.85
    sync percentiles (usec):
     |  1.00th=[  196],  5.00th=[  217], 10.00th=[  231], 20.00th=[  258],
     | 30.00th=[  302], 40.00th=[  351], 50.00th=[  392], 60.00th=[  453],
     | 70.00th=[  750], 80.00th=[ 1057], 90.00th=[ 1237], 95.00th=[ 1336],
     | 99.00th=[ 1598], 99.50th=[ 1778], 99.90th=[ 2606], 99.95th=[ 4883],
     | 99.99th=[14484]
  cpu          : usr=2.62%, sys=10.06%, ctx=258205, majf=0, minf=14
  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=47503,47219,0,94718 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=3167KiB/s (3243kB/s), 3167KiB/s-3167KiB/s (3243kB/s-3243kB/s), io=186MiB (195MB), run=60001-60001msec
  WRITE: bw=3148KiB/s (3223kB/s), 3148KiB/s-3148KiB/s (3223kB/s-3223kB/s), io=184MiB (193MB), run=60001-60001msec
```

</details>

#### 1024 read only test

```
$ fio --name TEST --eta-newline=5s --filename=temp.file \
--rw=read --size=500m --io_size=10g --blocksize=1024k \
--ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 \
--numjobs=1 --runtime=60 --group_reporting
[...]
  read: IOPS=2442, BW=2443MiB/s (2561MB/s)(10.0GiB/4192msec)
    slat (usec): min=39, max=1583, avg=149.53, stdev=82.49
    clat (usec): min=832, max=50823, avg=12705.42, stdev=4936.54
     lat (usec): min=972, max=50974, avg=12854.95, stdev=4940.71
```

<details>
<summary>full output</summary>

```
$ fio --name TEST --eta-newline=5s --filename=temp.file \
--rw=read --size=500m --io_size=10g --blocksize=1024k \
--ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 \
--numjobs=1 --runtime=60 --group_reporting
TEST: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=2441MiB/s][r=2441 IOPS][eta 00m:00s]
TEST: (groupid=0, jobs=1): err= 0: pid=5184: Wed Feb 21 14:01:57 2024
  read: IOPS=2442, BW=2443MiB/s (2561MB/s)(10.0GiB/4192msec)
    slat (usec): min=39, max=1583, avg=149.53, stdev=82.49
    clat (usec): min=832, max=50823, avg=12705.42, stdev=4936.54
     lat (usec): min=972, max=50974, avg=12854.95, stdev=4940.71
    clat percentiles (usec):
     |  1.00th=[ 1467],  5.00th=[ 3294], 10.00th=[ 4817], 20.00th=[ 7767],
     | 30.00th=[10421], 40.00th=[13173], 50.00th=[15401], 60.00th=[15664],
     | 70.00th=[15795], 80.00th=[15926], 90.00th=[16450], 95.00th=[17957],
     | 99.00th=[21890], 99.50th=[24511], 99.90th=[30540], 99.95th=[30802],
     | 99.99th=[50594]
   bw (  MiB/s): min= 2240, max= 2550, per=99.94%, avg=2441.25, stdev=97.56, samples=8
   iops        : min= 2240, max= 2550, avg=2441.25, stdev=97.56, samples=8
  lat (usec)   : 1000=0.22%
  lat (msec)   : 2=1.93%, 4=4.58%, 10=21.68%, 20=68.79%, 50=2.76%
  lat (msec)   : 100=0.03%
  cpu          : usr=2.12%, sys=38.80%, ctx=6386, majf=0, minf=8202
  IO depths    : 1=0.2%, 2=0.4%, 4=0.8%, 8=1.6%, 16=3.3%, 32=93.6%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.0%, 32=0.2%, 64=0.0%, >=64=0.0%
     issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=2443MiB/s (2561MB/s), 2443MiB/s-2443MiB/s (2561MB/s-2561MB/s), io=10.0GiB (10.7GB), run=4192-4192msec
```
</details>

# Raspberry PI 5 on SSD

#### 4k randrw test

```
$ fio --name TEST --eta-newline=5s --filename=temp.file --rw=randrw \
--size=500m --io_size=10g --blocksize=4k --ioengine=libaio --fsync=1 \
--iodepth=1 --direct=1 --numjobs=1 --runtime=60 --group_reporting
[...]
  read: IOPS=527, BW=2110KiB/s (2160kB/s)(124MiB/60001msec)
    slat (usec): min=9, max=464, avg=19.47, stdev= 6.89
    clat (usec): min=3, max=16846, avg=180.23, stdev=132.30
     lat (usec): min=141, max=16866, avg=199.70, stdev=132.51
  write: IOPS=528, BW=2113KiB/s (2164kB/s)(124MiB/60001msec); 0 zone resets
    slat (usec): min=10, max=695, avg=21.82, stdev=10.01
    clat (nsec): min=1648, max=7468.6k, avg=102478.22, stdev=52280.64
     lat (usec): min=80, max=7494, avg=124.30, stdev=55.32
```
<details>
    
<summary>full output</summary>

```
$ fio --name TEST --eta-newline=5s --filename=temp.file --rw=randrw \
--size=500m --io_size=10g --blocksize=4k --ioengine=libaio --fsync=1 \
--iodepth=1 --direct=1 --numjobs=1 --runtime=60 --group_reporting

TEST: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.33
Starting 1 process
TEST: Laying out IO file (1 file / 500MiB)
Jobs: 1 (f=1): [m(1)][11.7%][r=2134KiB/s,w=2062KiB/s][r=533,w=515 IOPS][eta 00m:53s]
Jobs: 1 (f=1): [m(1)][21.7%][r=2098KiB/s,w=2074KiB/s][r=524,w=518 IOPS][eta 00m:47s]
Jobs: 1 (f=1): [m(1)][31.7%][r=2076KiB/s,w=2096KiB/s][r=519,w=524 IOPS][eta 00m:41s]
Jobs: 1 (f=1): [m(1)][41.7%][r=1956KiB/s,w=2160KiB/s][r=489,w=540 IOPS][eta 00m:35s]
Jobs: 1 (f=1): [m(1)][51.7%][r=2018KiB/s,w=2202KiB/s][r=504,w=550 IOPS][eta 00m:29s]
Jobs: 1 (f=1): [m(1)][61.7%][r=1916KiB/s,w=2028KiB/s][r=479,w=507 IOPS][eta 00m:23s]
Jobs: 1 (f=1): [m(1)][71.7%][r=2348KiB/s,w=2016KiB/s][r=587,w=504 IOPS][eta 00m:17s]
Jobs: 1 (f=1): [m(1)][81.7%][r=2170KiB/s,w=2070KiB/s][r=542,w=517 IOPS][eta 00m:11s]
Jobs: 1 (f=1): [m(1)][91.7%][r=2098KiB/s,w=2230KiB/s][r=524,w=557 IOPS][eta 00m:05s]
Jobs: 1 (f=1): [m(1)][100.0%][r=2038KiB/s,w=2126KiB/s][r=509,w=531 IOPS][eta 00m:00s]
TEST: (groupid=0, jobs=1): err= 0: pid=1198248: Wed Feb 21 11:16:41 2024
  read: IOPS=527, BW=2110KiB/s (2160kB/s)(124MiB/60001msec)
    slat (usec): min=9, max=464, avg=19.47, stdev= 6.89
    clat (usec): min=3, max=16846, avg=180.23, stdev=132.30
     lat (usec): min=141, max=16866, avg=199.70, stdev=132.51
    clat percentiles (usec):
     |  1.00th=[  165],  5.00th=[  172], 10.00th=[  174], 20.00th=[  176],
     | 30.00th=[  178], 40.00th=[  178], 50.00th=[  180], 60.00th=[  180],
     | 70.00th=[  182], 80.00th=[  182], 90.00th=[  184], 95.00th=[  186],
     | 99.00th=[  198], 99.50th=[  212], 99.90th=[  383], 99.95th=[  519],
     | 99.99th=[  922]
   bw (  KiB/s): min= 1808, max= 2472, per=100.00%, avg=2112.00, stdev=136.22, samples=119
   iops        : min=  452, max=  618, avg=528.00, stdev=34.06, samples=119
  write: IOPS=528, BW=2113KiB/s (2164kB/s)(124MiB/60001msec); 0 zone resets
    slat (usec): min=10, max=695, avg=21.82, stdev=10.01
    clat (nsec): min=1648, max=7468.6k, avg=102478.22, stdev=52280.64
     lat (usec): min=80, max=7494, avg=124.30, stdev=55.32
    clat percentiles (usec):
     |  1.00th=[   70],  5.00th=[   71], 10.00th=[   72], 20.00th=[   73],
     | 30.00th=[   74], 40.00th=[  114], 50.00th=[  118], 60.00th=[  119],
     | 70.00th=[  121], 80.00th=[  122], 90.00th=[  123], 95.00th=[  125],
     | 99.00th=[  135], 99.50th=[  147], 99.90th=[  208], 99.95th=[  314],
     | 99.99th=[  685]
   bw (  KiB/s): min= 1920, max= 2296, per=100.00%, avg=2114.15, stdev=73.79, samples=119
   iops        : min=  480, max=  574, avg=528.54, stdev=18.45, samples=119
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.04%
  lat (usec)   : 100=19.21%, 250=80.59%, 500=0.10%, 750=0.03%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
  fsync/fdatasync/sync_file_range:
    sync (usec): min=197, max=26452, avg=925.15, stdev=760.17
    sync percentiles (usec):
     |  1.00th=[  338],  5.00th=[  347], 10.00th=[  351], 20.00th=[  355],
     | 30.00th=[  355], 40.00th=[  359], 50.00th=[  416], 60.00th=[  474],
     | 70.00th=[ 1467], 80.00th=[ 2024], 90.00th=[ 2089], 95.00th=[ 2089],
     | 99.00th=[ 2147], 99.50th=[ 2180], 99.90th=[ 3064], 99.95th=[ 3064],
     | 99.99th=[ 8356]
  cpu          : usr=0.61%, sys=2.87%, ctx=182526, majf=0, minf=9
  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=31647,31700,0,63343 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=2110KiB/s (2160kB/s), 2110KiB/s-2110KiB/s (2160kB/s-2160kB/s), io=124MiB (130MB), run=60001-60001msec
  WRITE: bw=2113KiB/s (2164kB/s), 2113KiB/s-2113KiB/s (2164kB/s-2164kB/s), io=124MiB (130MB), run=60001-60001msec

Disk stats (read/write):
  sda: ios=31601/109919, merge=27/14981, ticks=6110/52656, in_queue=104054, util=99.89%
```
</details>

#### 1024k read only test

```
$ fio --name TEST --eta-newline=5s --filename=temp.file \
--rw=read --size=500m --io_size=10g --blocksize=1024k \
--ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 \
--numjobs=1 --runtime=60 --group_reporting
[...]
  read: IOPS=375, BW=375MiB/s (393MB/s)(10.0GiB/27287msec)
    slat (usec): min=6, max=19892, avg=2500.21, stdev=3198.07
    clat (msec): min=8, max=160, avg=82.50, stdev=15.20
     lat (msec): min=8, max=163, avg=85.00, stdev=15.49
```
<details>
<summary>full output</summary>

```
$ fio --name TEST --eta-newline=5s --filename=temp.file \
--rw=read --size=500m --io_size=10g --blocksize=1024k \
--ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 \
--numjobs=1 --runtime=60 --group_reporting
TEST: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [R(1)][25.9%][r=375MiB/s][r=375 IOPS][eta 00m:20s]
Jobs: 1 (f=1): [R(1)][48.1%][r=374MiB/s][r=374 IOPS][eta 00m:14s]
Jobs: 1 (f=1): [R(1)][70.4%][r=377MiB/s][r=377 IOPS][eta 00m:08s]
Jobs: 1 (f=1): [R(1)][92.6%][r=376MiB/s][r=376 IOPS][eta 00m:02s]
Jobs: 1 (f=1): [R(1)][100.0%][r=375MiB/s][r=375 IOPS][eta 00m:00s]
TEST: (groupid=0, jobs=1): err= 0: pid=1202561: Wed Feb 21 13:13:55 2024
  read: IOPS=375, BW=375MiB/s (393MB/s)(10.0GiB/27287msec)
    slat (usec): min=6, max=19892, avg=2500.21, stdev=3198.07
    clat (msec): min=8, max=160, avg=82.50, stdev=15.20
     lat (msec): min=8, max=163, avg=85.00, stdev=15.49
    clat percentiles (msec):
     |  1.00th=[   21],  5.00th=[   73], 10.00th=[   81], 20.00th=[   81],
     | 30.00th=[   81], 40.00th=[   81], 50.00th=[   81], 60.00th=[   87],
     | 70.00th=[   87], 80.00th=[   87], 90.00th=[   87], 95.00th=[   93],
     | 99.00th=[  146], 99.50th=[  153], 99.90th=[  161], 99.95th=[  161],
     | 99.99th=[  161]
   bw (  KiB/s): min=374784, max=393216, per=100.00%, avg=384417.19, stdev=2687.39, samples=54
   iops        : min=  366, max=  384, avg=375.41, stdev= 2.62, samples=54
  lat (msec)   : 10=0.20%, 20=0.64%, 50=2.46%, 100=92.27%, 250=4.43%
  cpu          : usr=0.15%, sys=1.00%, ctx=5772, majf=0, minf=2056
  IO depths    : 1=0.2%, 2=0.4%, 4=0.8%, 8=1.6%, 16=3.3%, 32=93.6%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.0%, 32=0.2%, 64=0.0%, >=64=0.0%
     issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=375MiB/s (393MB/s), 375MiB/s-375MiB/s (393MB/s-393MB/s), io=10.0GiB (10.7GB), run=27287-27287msec

Disk stats (read/write):
  sda: ios=20373/8, merge=0/16, ticks=1520895/4391, in_queue=1525409, util=99.69%
```
</details>
