# np04-srv-004 RAID reconfiguration
###### tags: `DUNE-DAQ`
## Existing RAID layout and available disks
### Existing configuration
```shell=
[root@np04-srv-004 ~]# mdadm --verbose --detail -scan | tee mdadm.conf.2021.10.01
ARRAY /dev/md/0 level=raid5 num-devices=11 metadata=1.2 spares=1 name=np04-srv-004:0 UUID=96bab78f:db437cbe:d9912422:9cfb314f
devices=/dev/sda,/dev/sdb,/dev/sdc,/dev/sdd,/dev/sde,/dev/sdf,/dev/sdg,/dev/sdh,/dev/sdi,/dev/sdn,/dev/sdy,/dev/sdz
ARRAY /dev/md/1 level=raid5 num-devices=11 metadata=1.2 spares=1 name=np04-srv-004:1 UUID=3f3543de:0a35ebdb:fc3ff03f:0033179d
devices=/dev/sdaa,/dev/sdab,/dev/sdac,/dev/sdad,/dev/sdae,/dev/sdaf,/dev/sdag,/dev/sdk,/dev/sdl,/dev/sdm,/dev/sdp,/dev/sdq
ARRAY /dev/md/2 level=raid5 num-devices=11 metadata=1.2 spares=1 name=np04-srv-004:2 UUID=bf465668:51771dcd:70d9053f:e18e2725
devices=/dev/sdah,/dev/sdai,/dev/sdaj,/dev/sdak,/dev/sdal,/dev/sdam,/dev/sdan,/dev/sdao,/dev/sdap,/dev/sdj,/dev/sdo,/dev/sdr
ARRAY /dev/md/3 level=raid5 num-devices=11 metadata=1.2 spares=1 name=np04-srv-004:3 UUID=41fe515a:816e967a:185c239d:5ecbe9dd
devices=/dev/sdaq,/dev/sdar,/dev/sdas,/dev/sdat,/dev/sdau,/dev/sdav,/dev/sds,/dev/sdt,/dev/sdu,/dev/sdv,/dev/sdw,/dev/sdx
...
```
### Disk specs
Looks like each disk has the same specification as others. Here is one for example. Each has a size of 3.0TB.
```shell=
[root@np04-srv-004 ~]# mdadm -E /dev/sda
/dev/sda:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 96bab78f:db437cbe:d9912422:9cfb314f
Name : np04-srv-004:0 (local to host np04-srv-004)
Creation Time : Mon Jul 9 11:19:06 2018
Raid Level : raid5
Raid Devices : 11
Avail Dev Size : 5860271024 (2794.39 GiB 3000.46 GB)
Array Size : 29301350400 (27943.95 GiB 30004.58 GB)
Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=944 sectors
State : clean
Device UUID : 4b36a3e0:dcb34ac9:8bc7e741:b0266493
Internal Bitmap : 8 sectors from superblock
Update Time : Sat Oct 2 21:18:21 2021
Bad Block Log : 512 entries available at offset 24 sectors
Checksum : 3de2a3a5 - correct
Events : 35582
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
[root@np04-srv-004 raid_reconfig]# fdisk -l /dev/sdak
Disk /dev/sdak: 3000.6 GB, 3000592982016 bytes, 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
```
### List of Disks
The list of disks are saved in the following directory. There are total 48 disks, with 12 disks for each previous raid arrays.
```=
[root@np04-srv-004 ~]# tree /root/raid_reconfig/
/root/raid_reconfig/
├── all.list
├── data0.list
├── data1.list
├── data2.list
└── data3.list
0 directories, 5 files
```
## Reconfigure RAID
* Umount and stop current RAID arrays
```shell=
for i in `seq 0 3`; do umount /data$i; mdadm --stop /dev/md$i; done
```
* Clear current raid configuration
```shell=
while read p; do echo $p; mdadm --zero-superblock $p; done < /root/raid_reconfig/all.list
```
### Create new raid arrays
* Assemble `mdadm` commands
```shell=
[root@np04-srv-004 raid_reconfig]# for i in `seq 0 3`; do md_list=""; while read p; do md_list="${md_list} $p"; done < /root/raid_reconfig/data${i}.list; echo "mdadm --create --verbose /dev/md$i --level=10 --raid-devices=12 $md_list"; done
mdadm --create --verbose /dev/md0 --level=10 --raid-devices=12 /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdn /dev/sdy /dev/sdz
mdadm --create --verbose /dev/md1 --level=10 --raid-devices=12 /dev/sdaa /dev/sdab /dev/sdac /dev/sdad /dev/sdae /dev/sdaf /dev/sdag /dev/sdk /dev/sdl /dev/sdm /dev/sdp /dev/sdq
mdadm --create --verbose /dev/md2 --level=10 --raid-devices=12 /dev/sdah /dev/sdai /dev/sdaj /dev/sdak /dev/sdal /dev/sdam /dev/sdan /dev/sdao /dev/sdap /dev/sdj /dev/sdo /dev/sdr
mdadm --create --verbose /dev/md3 --level=10 --raid-devices=12 /dev/sdaq /dev/sdar /dev/sdas /dev/sdat /dev/sdau /dev/sdav /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx
```
* Execute `mdadm` commands
```shell=
[root@np04-srv-004 raid_reconfig]# mdadm --create --verbose /dev/md0 --level=10 --raid-devices=12 /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdn /dev/sdy /dev/sdz
mdadm: layout defaults to n2
mdadm: layout defaults to n2
mdadm: chunk size defaults to 512K
mdadm: size set to 2930134016K
mdadm: automatically enabling write-intent bitmap on large array
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.
[root@np04-srv-004 raid_reconfig]# mdadm --create --verbose /dev/md1 --level=10 --raid-devices=12 /dev/sdaa /dev/sdab /dev/sdac /dev/sdad /dev/sdae /dev/sdaf /dev/sdag /dev/sdk /dev/sdl /dev/sdm /dev/sdp /dev/sdq
mdadm: layout defaults to n2
mdadm: layout defaults to n2
mdadm: chunk size defaults to 512K
mdadm: size set to 2930134016K
mdadm: automatically enabling write-intent bitmap on large array
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md1 started.
[root@np04-srv-004 raid_reconfig]# mdadm --create --verbose /dev/md2 --level=10 --raid-devices=12 /dev/sdah /dev/sdai /dev/sdaj /dev/sdak /dev/sdal /dev/sdam /dev/sdan /dev/sdao /dev/sdap /dev/sdj /dev/sdo /dev/sdr
mdadm: layout defaults to n2
mdadm: layout defaults to n2
mdadm: chunk size defaults to 512K
mdadm: size set to 2930134016K
mdadm: automatically enabling write-intent bitmap on large array
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md2 started.
[root@np04-srv-004 raid_reconfig]# mdadm --create --verbose /dev/md3 --level=10 --raid-devices=12 /dev/sdaq /dev/sdar /dev/sdas /dev/sdat /dev/sdau /dev/sdav /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx
mdadm: layout defaults to n2
mdadm: layout defaults to n2
mdadm: chunk size defaults to 512K
mdadm: size set to 2930134016K
mdadm: automatically enabling write-intent bitmap on large array
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md3 started.
```
* Check raid configuration
```shell=
[root@np04-srv-004 raid_reconfig]# mdadm -E /dev/sda
/dev/sda:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : ad8a9652:6a395265:33d5ada3:f91bfc83
Name : np04-srv-004:0 (local to host np04-srv-004)
Creation Time : Sun Oct 3 06:03:05 2021
Raid Level : raid10
Raid Devices : 12
Avail Dev Size : 5860268976 (2794.39 GiB 3000.46 GB)
Array Size : 17580804096 (16766.36 GiB 18002.74 GB)
Used Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Data Offset : 264192 sectors
Super Offset : 8 sectors
Unused Space : before=264096 sectors, after=944 sectors
State : active
Device UUID : e2cbaf5d:f9c2211b:60c35ee7:0153ff83
Internal Bitmap : 8 sectors from superblock
Update Time : Sun Oct 3 06:08:01 2021
Bad Block Log : 512 entries available at offset 80 sectors
Checksum : c9028765 - correct
Events : 56
Layout : near=2
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
```
* Create file system, and mount the arrays
```shell=
[root@np04-srv-004 raid_reconfig]# for i in `seq 0 3`; do mkfs.xfs /dev/md$i; done
meta-data=/dev/md0 isize=512 agcount=32, agsize=137350144 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0, sparse=0
data = bsize=4096 blocks=4395201024, imaxpct=5
= sunit=128 swidth=768 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=521728, version=2
= sectsz=512 sunit=8 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
meta-data=/dev/md1 isize=512 agcount=32, agsize=137350144 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0, sparse=0
data = bsize=4096 blocks=4395201024, imaxpct=5
= sunit=128 swidth=768 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=521728, version=2
= sectsz=512 sunit=8 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
meta-data=/dev/md2 isize=512 agcount=32, agsize=137350144 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0, sparse=0
data = bsize=4096 blocks=4395201024, imaxpct=5
= sunit=128 swidth=768 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=521728, version=2
= sectsz=512 sunit=8 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
meta-data=/dev/md3 isize=512 agcount=32, agsize=137350144 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0, sparse=0
data = bsize=4096 blocks=4395201024, imaxpct=5
= sunit=128 swidth=768 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=521728, version=2
= sectsz=512 sunit=8 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
[root@np04-srv-004 raid_reconfig]# for i in `seq 0 3`; do mount /dev/md$i /data$i; done
mount: /dev/md0 is already mounted or /data0 busy
/dev/md0 is already mounted on /data0
mount: /dev/md1 is already mounted or /data1 busy
/dev/md1 is already mounted on /data1
mount: /dev/md2 is already mounted or /data2 busy
/dev/md2 is already mounted on /data2
mount: /dev/md3 is already mounted or /data3 busy
/dev/md3 is already mounted on /data3
[root@np04-srv-004 raid_reconfig]# mount|grep data
/dev/md0 on /data0 type xfs (rw,relatime,attr2,inode64,sunit=1024,swidth=6144,noquota)
/dev/md1 on /data1 type xfs (rw,relatime,attr2,inode64,sunit=1024,swidth=6144,noquota)
/dev/md2 on /data2 type xfs (rw,relatime,attr2,inode64,sunit=1024,swidth=6144,noquota)
/dev/md3 on /data3 type xfs (rw,relatime,attr2,inode64,sunit=1024,swidth=6144,noquota)
```
* Update /etc/mdadm.conf
```shell=
[root@np04-srv-004 raid_reconfig]# mv /etc/mdadm.conf /etc/mdadm.conf.raid5
[root@np04-srv-004 raid_reconfig]# mdadm --detail -scan > /etc/mdadm.conf
```
* new raid array sizes is 17TB each; previously it was 28TB.
```shell=
/dev/md0 17T 34M 17T 1% /data0
/dev/md1 17T 34M 17T 1% /data1
/dev/md2 17T 34M 17T 1% /data2
/dev/md3 17T 34M 17T 1% /data3
```
## Benchmarking new raid arrays
* `dd` on raid 10@np04-srv-004
```shell=
[root@np04-srv-004 ~]# dd if=/data2/bench_mark.trash of=/data1/dd_copy.trash bs=1G count=1 conv=fdatasync
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 11.1652 s, 96.2 MB/s
```
* `dd` on raid 5@np04-srv-003
```shell=
[root@np04-srv-003 ~]# dd if=/data0/bench_mark.trash of=/data1/dd_copy.trash bs=1G count=1 conv=fdatasync
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 8.43239 s, 127 MB/s
```
* `dd` on raid10@np04-srv-003
```shell=
[pding@np04-srv-003 data0]$ dd if=/data0/1GB.bin of=/data1/dd_copy.trash bs=1G count=1 conv=fdatasync
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.64865 s, 231 MB/s
```
* Simple test with `fio`
```shell=
[root@np04-srv-004 pding]# fio --name=seqwrite --rw=write --direct=1 --ioengine=libaio --bs=4k --numjobs=4 --size=4G --runtime=60 --filename=/data1/testfile
seqwrite: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.1
Starting 4 processes
seqwrite: Laying out IO file (1 file / 4096MiB)
Jobs: 4 (f=4): [W(4)][100.0%][r=0KiB/s,w=624KiB/s][r=0,w=156 IOPS][eta 00m:00s]
seqwrite: (groupid=0, jobs=1): err= 0: pid=9649: Sun Oct 3 06:41:21 2021
write: IOPS=55, BW=222KiB/s (228kB/s)(13.0MiB/60003msec)
slat (usec): min=9, max=313268, avg=12444.35, stdev=17162.59
clat (usec): min=109, max=128906, avg=5550.17, stdev=4846.79
lat (usec): min=378, max=326166, avg=17995.07, stdev=18494.12
clat percentiles (usec):
| 1.00th=[ 269], 5.00th=[ 586], 10.00th=[ 914], 20.00th=[ 1844],
| 30.00th=[ 2704], 40.00th=[ 3556], 50.00th=[ 4490], 60.00th=[ 5538],
| 70.00th=[ 6915], 80.00th=[ 8586], 90.00th=[ 11731], 95.00th=[ 14091],
| 99.00th=[ 17957], 99.50th=[ 20317], 99.90th=[ 32900], 99.95th=[ 45876],
| 99.99th=[128451]
bw ( KiB/s): min= 88, max= 640, per=25.00%, avg=222.21, stdev=83.49, samples=120
iops : min= 22, max= 160, avg=55.55, stdev=20.86, samples=120
lat (usec) : 250=0.51%, 500=3.12%, 750=3.54%, 1000=3.90%
lat (msec) : 2=11.13%, 4=23.40%, 10=39.95%, 20=13.92%, 50=0.51%
lat (msec) : 250=0.03%
cpu : usr=0.04%, sys=0.20%, ctx=13006, majf=0, minf=30
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,3334,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
seqwrite: (groupid=0, jobs=1): err= 0: pid=9650: Sun Oct 3 06:41:21 2021
write: IOPS=55, BW=222KiB/s (228kB/s)(13.0MiB/60006msec)
slat (usec): min=9, max=313829, avg=12405.35, stdev=17402.95
clat (usec): min=97, max=37194, avg=5579.39, stdev=4499.11
lat (usec): min=114, max=321986, avg=17985.30, stdev=18640.16
clat percentiles (usec):
| 1.00th=[ 260], 5.00th=[ 619], 10.00th=[ 898], 20.00th=[ 1729],
| 30.00th=[ 2507], 40.00th=[ 3458], 50.00th=[ 4555], 60.00th=[ 5604],
| 70.00th=[ 7046], 80.00th=[ 8848], 90.00th=[11863], 95.00th=[14615],
| 99.00th=[19006], 99.50th=[21365], 99.90th=[29754], 99.95th=[33162],
| 99.99th=[36963]
bw ( KiB/s): min= 96, max= 624, per=25.01%, avg=222.33, stdev=83.47, samples=120
iops : min= 24, max= 156, avg=55.58, stdev=20.87, samples=120
lat (usec) : 100=0.06%, 250=0.69%, 500=2.79%, 750=3.75%, 1000=4.17%
lat (msec) : 2=11.90%, 4=21.01%, 10=40.02%, 20=14.96%, 50=0.66%
cpu : usr=0.03%, sys=0.22%, ctx=12983, majf=0, minf=28
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,3336,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
seqwrite: (groupid=0, jobs=1): err= 0: pid=9651: Sun Oct 3 06:41:21 2021
write: IOPS=55, BW=222KiB/s (228kB/s)(13.0MiB/60009msec)
slat (usec): min=10, max=310567, avg=12063.85, stdev=17311.77
clat (usec): min=131, max=31311, avg=5916.47, stdev=4720.27
lat (usec): min=379, max=313818, avg=17980.87, stdev=18418.71
clat percentiles (usec):
| 1.00th=[ 273], 5.00th=[ 611], 10.00th=[ 947], 20.00th=[ 1876],
| 30.00th=[ 2802], 40.00th=[ 3687], 50.00th=[ 4686], 60.00th=[ 5866],
| 70.00th=[ 7242], 80.00th=[ 9372], 90.00th=[13042], 95.00th=[15533],
| 99.00th=[20055], 99.50th=[22676], 99.90th=[27919], 99.95th=[29754],
| 99.99th=[31327]
bw ( KiB/s): min= 104, max= 608, per=25.02%, avg=222.41, stdev=82.94, samples=120
iops : min= 26, max= 152, avg=55.60, stdev=20.73, samples=120
lat (usec) : 250=0.51%, 500=3.21%, 750=3.36%, 1000=3.72%
lat (msec) : 2=10.52%, 4=21.46%, 10=39.41%, 20=16.87%, 50=0.96%
cpu : usr=0.03%, sys=0.22%, ctx=12571, majf=0, minf=27
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,3337,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
seqwrite: (groupid=0, jobs=1): err= 0: pid=9652: Sun Oct 3 06:41:21 2021
write: IOPS=55, BW=222KiB/s (228kB/s)(13.0MiB/60021msec)
slat (usec): min=9, max=309272, avg=12427.00, stdev=17294.02
clat (usec): min=126, max=30949, avg=5567.78, stdev=4470.77
lat (usec): min=289, max=319413, avg=17995.36, stdev=18575.08
clat percentiles (usec):
| 1.00th=[ 273], 5.00th=[ 553], 10.00th=[ 938], 20.00th=[ 1811],
| 30.00th=[ 2638], 40.00th=[ 3425], 50.00th=[ 4424], 60.00th=[ 5538],
| 70.00th=[ 6783], 80.00th=[ 8717], 90.00th=[12256], 95.00th=[14746],
| 99.00th=[19268], 99.50th=[20317], 99.90th=[26870], 99.95th=[30802],
| 99.99th=[31065]
bw ( KiB/s): min= 104, max= 608, per=25.00%, avg=222.28, stdev=82.74, samples=120
iops : min= 26, max= 152, avg=55.57, stdev=20.68, samples=120
lat (usec) : 250=0.39%, 500=4.02%, 750=3.18%, 1000=3.27%
lat (msec) : 2=11.75%, 4=23.12%, 10=38.83%, 20=14.81%, 50=0.63%
cpu : usr=0.03%, sys=0.21%, ctx=12976, majf=0, minf=28
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,3335,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=889KiB/s (910kB/s), 222KiB/s-222KiB/s (228kB/s-228kB/s), io=52.1MiB (54.6MB), run=60003-60021msec
Disk stats (read/write):
md1: ios=0/13334, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=593/2557, aggrmerge=221/97, aggrticks=3563/13516, aggrin_queue=17261, aggrutil=21.65%
sdk: ios=594/3183, merge=224/185, ticks=4636/22078, in_queue=26895, util=21.65%
sdl: ios=592/2120, merge=225/0, ticks=3385/12198, in_queue=15819, util=13.33%
sdm: ios=593/2702, merge=225/184, ticks=4754/16150, in_queue=21093, util=18.04%
sdp: ios=594/2090, merge=224/0, ticks=2918/14747, in_queue=17919, util=14.48%
sdq: ios=594/2670, merge=224/193, ticks=5198/18111, in_queue=23572, util=19.87%
sdaa: ios=592/2093, merge=220/2, ticks=2246/9193, in_queue=11586, util=9.69%
sdab: ios=594/2676, merge=220/199, ticks=4545/11786, in_queue=16503, util=13.21%
sdac: ios=594/2091, merge=219/0, ticks=2166/9848, in_queue=12116, util=10.09%
sdad: ios=594/2675, merge=219/204, ticks=4093/12300, in_queue=16546, util=13.90%
sdae: ios=593/2605, merge=220/0, ticks=2760/10934, in_queue=13814, util=11.74%
sdaf: ios=593/3185, merge=220/205, ticks=3973/13231, in_queue=17373, util=14.34%
sdag: ios=593/2602, merge=217/0, ticks=2088/11619, in_queue=13901, util=11.37%
[
```
* The speed with no tuning of the configuration seems awful. But this might due to the small block size used in the test. Using a much bigger block size value shows the raid 10 out-performs raid 5 (218MB/s vs 61.3MB/s with `bs=1024k`.
```shell=
[root@np04-srv-004 ~]# fio --name=seqwrite --rw=write --direct=1 --ioengine=libaio --bs=1024k --numjobs=10 --size=40G --runtime=60 --filename=/data3/bench_mark.trash --group_reporting=1
seqwrite: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
...
fio-3.1
Starting 10 processes
Jobs: 10 (f=10): [W(10)][100.0%][r=0KiB/s,w=213MiB/s][r=0,w=213 IOPS][eta 00m:00s]
seqwrite: (groupid=0, jobs=10): err= 0: pid=32600: Sun Oct 3 08:28:43 2021
write: IOPS=207, BW=208MiB/s (218MB/s)(12.2GiB/60055msec)
slat (usec): min=841, max=245292, avg=39368.11, stdev=23949.42
clat (usec): min=1314, max=50703, avg=8757.39, stdev=5257.18
lat (msec): min=5, max=281, avg=48.13, stdev=24.37
clat percentiles (usec):
| 1.00th=[ 1991], 5.00th=[ 2671], 10.00th=[ 3163], 20.00th=[ 3949],
| 30.00th=[ 5080], 40.00th=[ 6456], 50.00th=[ 7701], 60.00th=[ 9110],
| 70.00th=[10683], 80.00th=[12780], 90.00th=[15926], 95.00th=[18744],
| 99.00th=[24773], 99.50th=[27657], 99.90th=[33424], 99.95th=[39060],
| 99.99th=[49021]
bw ( KiB/s): min= 8208, max=32899, per=10.01%, avg=21298.02, stdev=4115.36, samples=1200
iops : min= 8, max= 32, avg=20.77, stdev= 4.02, samples=1200
lat (msec) : 2=1.03%, 4=19.38%, 10=45.02%, 20=31.02%, 50=3.54%
lat (msec) : 100=0.01%
cpu : usr=0.14%, sys=0.40%, ctx=112110, majf=0, minf=308
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,12475,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=208MiB/s (218MB/s), 208MiB/s-208MiB/s (218MB/s-218MB/s), io=12.2GiB (13.1GB), run=60055-60055msec
Disk stats (read/write):
md3: ios=0/24893, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=455/4446, aggrmerge=162/83, aggrticks=3334/35097, aggrin_queue=38202, aggrutil=30.85%
sds: ios=454/4228, merge=159/0, ticks=1906/37164, in_queue=39258, util=28.69%
sdt: ios=459/4686, merge=164/172, ticks=6279/39730, in_queue=45165, util=30.55%
sdu: ios=447/4219, merge=164/0, ticks=1848/34357, in_queue=36333, util=26.96%
sdv: ios=461/4675, merge=165/197, ticks=5378/37802, in_queue=41470, util=27.91%
sdw: ios=461/4219, merge=156/0, ticks=1848/35261, in_queue=37316, util=27.60%
sdx: ios=467/4675, merge=159/182, ticks=6281/40693, in_queue=46255, util=30.85%
sdaq: ios=452/4217, merge=160/0, ticks=2174/32358, in_queue=34631, util=26.33%
sdar: ios=463/4678, merge=163/164, ticks=5306/36270, in_queue=40800, util=27.84%
sdas: ios=450/4217, merge=159/0, ticks=1578/29300, in_queue=31301, util=23.22%
sdat: ios=456/4656, merge=159/134, ticks=3171/32138, in_queue=35633, util=26.12%
sdau: ios=441/4225, merge=171/0, ticks=1396/31568, in_queue=33085, util=24.66%
sdav: ios=449/4668, merge=171/156, ticks=2851/34526, in_queue=37181, util=26.65%
[root@np04-srv-003 ~]# fio --name=seqwrite --rw=write --direct=1 --ioengine=libaio --bs=1024k --numjobs=10 --size=40G --runtime=60 --filename=/data3/bench_mark.trash --group_reporting=1
seqwrite: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
...
fio-3.1
Starting 10 processes
seqwrite: Laying out IO file (1 file / 40960MiB)
Jobs: 10 (f=10): [W(10)][100.0%][r=0KiB/s,w=74.0MiB/s][r=0,w=74 IOPS][eta 00m:00s]
seqwrite: (groupid=0, jobs=10): err= 0: pid=147567: Sun Oct 3 08:29:08 2021
write: IOPS=58, BW=58.4MiB/s (61.3MB/s)(3520MiB/60247msec)
slat (usec): min=131, max=820207, avg=49771.01, stdev=93800.68
clat (usec): min=1383, max=1490.3k, avg=121045.18, stdev=96095.60
lat (msec): min=4, max=1503, avg=170.82, stdev=129.43
clat percentiles (msec):
| 1.00th=[ 6], 5.00th=[ 12], 10.00th=[ 19], 20.00th=[ 43],
| 30.00th=[ 65], 40.00th=[ 83], 50.00th=[ 101], 60.00th=[ 123],
| 70.00th=[ 148], 80.00th=[ 186], 90.00th=[ 243], 95.00th=[ 300],
| 99.00th=[ 443], 99.50th=[ 502], 99.90th=[ 625], 99.95th=[ 625],
| 99.99th=[ 1485]
bw ( KiB/s): min= 2048, max=16416, per=10.17%, avg=6083.02, stdev=2820.00, samples=1183
iops : min= 2, max= 16, avg= 5.93, stdev= 2.75, samples=1183
lat (msec) : 2=0.09%, 4=0.28%, 10=3.35%, 20=6.93%, 50=12.41%
lat (msec) : 100=26.39%, 250=41.19%, 500=8.86%, 750=0.45%, 2000=0.03%
cpu : usr=0.02%, sys=0.27%, ctx=144494, majf=0, minf=280
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,3520,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=58.4MiB/s (61.3MB/s), 58.4MiB/s-58.4MiB/s (61.3MB/s-61.3MB/s), io=3520MiB (3691MB), run=60247-60247msec
Disk stats (read/write):
md3: ios=0/7047, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=3394/18123, aggrmerge=19966/82725, aggrticks=115228/217328, aggrin_queue=333246, aggrutil=40.60%
sdb: ios=3667/19965, merge=21623/91110, ticks=121792/247354, in_queue=369843, util=32.19%
sdh: ios=3803/20083, merge=21911/90432, ticks=113484/211946, in_queue=326112, util=30.19%
sdr: ios=3766/20026, merge=22081/91592, ticks=150530/293677, in_queue=445127, util=35.71%
sdw: ios=3666/19684, merge=20922/90848, ticks=128778/217018, in_queue=346285, util=30.70%
sdab: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
sdah: ios=3691/19341, merge=21500/89111, ticks=130627/218317, in_queue=349533, util=31.52%
sdaq: ios=3633/19466, merge=21246/88884, ticks=106500/218793, in_queue=326505, util=29.23%
sdar: ios=3718/19687, merge=21802/89309, ticks=105031/190426, in_queue=296320, util=28.49%
sdas: ios=3792/19511, merge=22651/88559, ticks=109159/233127, in_queue=343032, util=29.83%
sdat: ios=3682/19841, merge=22152/90256, ticks=120539/222859, in_queue=344145, util=29.89%
sdau: ios=3713/20182, merge=21858/91090, ticks=109683/191460, in_queue=301702, util=27.75%
sdav: ios=3602/19690, merge=21846/91516, ticks=186616/362970, in_queue=550355, util=40.60%
```
* using 4MiB block size, `305MB/s vs 115MB/s`. And ` WRITE: bw=376MiB/s (395MB/s), 376MiB/s-376MiB/s (395MB/s-395MB/s), io=22.1GiB (23.7GB), run=60014-60014msec` with RAID 10 on `np04-srv-003`.
```shell=
[root@np04-srv-004 ~]# fio --name=seqwrite --rw=write --direct=1 --ioengine=libaio --bs=4m --numjobs=10 --size=40G --runtime=60 --filename=/data3/bench_mark.trash --group_reporting=1
seqwrite: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=1
...
fio-3.1
Starting 10 processes
Jobs: 10 (f=10): [W(10)][100.0%][r=0KiB/s,w=200MiB/s][r=0,w=50 IOPS][eta 00m:00s]
seqwrite: (groupid=0, jobs=10): err= 0: pid=34106: Sun Oct 3 08:35:28 2021
write: IOPS=72, BW=291MiB/s (305MB/s)(17.1GiB/60197msec)
slat (msec): min=3, max=484, avg=111.53, stdev=69.42
clat (usec): min=1322, max=99474, avg=25816.35, stdev=20035.81
lat (msec): min=17, max=527, avg=137.35, stdev=57.49
clat percentiles (usec):
| 1.00th=[ 2089], 5.00th=[ 2900], 10.00th=[ 3523], 20.00th=[ 5145],
| 30.00th=[ 7832], 40.00th=[11600], 50.00th=[22152], 60.00th=[33162],
| 70.00th=[40633], 80.00th=[46400], 90.00th=[53216], 95.00th=[58459],
| 99.00th=[68682], 99.50th=[71828], 99.90th=[82314], 99.95th=[86508],
| 99.99th=[99091]
bw ( KiB/s): min= 8192, max=57458, per=10.08%, avg=30067.85, stdev=9680.98, samples=1193
iops : min= 2, max= 14, avg= 7.33, stdev= 2.36, samples=1193
lat (msec) : 2=0.78%, 4=13.21%, 10=22.46%, 20=12.12%, 50=36.76%
lat (msec) : 100=14.67%
cpu : usr=0.20%, sys=0.42%, ctx=100077, majf=0, minf=309
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,4382,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=291MiB/s (305MB/s), 291MiB/s-291MiB/s (305MB/s-305MB/s), io=17.1GiB (18.4GB), run=60197-60197msec
Disk stats (read/write):
md3: ios=0/35022, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=378/6105, aggrmerge=146/66, aggrticks=3514/94026, aggrin_queue=97041, aggrutil=41.70%
sds: ios=369/5922, merge=147/0, ticks=1446/111068, in_queue=112696, util=40.30%
sdt: ios=378/6297, merge=151/113, ticks=5779/111656, in_queue=116391, util=41.57%
sdu: ios=372/5916, merge=148/0, ticks=1764/81437, in_queue=83334, util=39.89%
sdv: ios=381/6296, merge=148/129, ticks=5280/85746, in_queue=90104, util=41.32%
sdw: ios=374/5916, merge=145/0, ticks=1557/109162, in_queue=110832, util=39.74%
sdx: ios=381/6290, merge=145/115, ticks=4098/113762, in_queue=117380, util=41.70%
sdaq: ios=374/5913, merge=146/0, ticks=1481/74715, in_queue=76348, util=37.12%
sdar: ios=375/6297, merge=155/149, ticks=5877/81550, in_queue=85548, util=39.10%
sdas: ios=387/5913, merge=135/0, ticks=1552/97988, in_queue=99747, util=35.86%
sdat: ios=397/6295, merge=141/146, ticks=6093/103893, in_queue=108880, util=38.06%
sdau: ios=371/5918, merge=145/0, ticks=1201/75673, in_queue=76952, util=36.24%
sdav: ios=386/6298, merge=150/148, ticks=6040/81671, in_queue=86288, util=39.83%
[root@np04-srv-003 ~]# fio --name=seqwrite --rw=write --direct=1 --ioengine=libaio --bs=4m --numjobs=10 --size=40G --runtime=60 --filename=/data3/bench_mark.trash --group_reporting=1
seqwrite: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=1
...
fio-3.1
Starting 10 processes
Jobs: 10 (f=10): [W(10)][100.0%][r=0KiB/s,w=68.0MiB/s][r=0,w=17 IOPS][eta 00m:00s]
seqwrite: (groupid=0, jobs=10): err= 0: pid=149435: Sun Oct 3 08:36:19 2021
write: IOPS=27, BW=109MiB/s (115MB/s)(6592MiB/60237msec)
slat (usec): min=541, max=1328.5k, avg=157431.29, stdev=206773.52
clat (usec): min=520, max=4433.3k, avg=207773.14, stdev=156061.64
lat (msec): min=17, max=4705, avg=365.21, stdev=274.14
clat percentiles (msec):
| 1.00th=[ 10], 5.00th=[ 65], 10.00th=[ 97], 20.00th=[ 121],
| 30.00th=[ 138], 40.00th=[ 157], 50.00th=[ 178], 60.00th=[ 207],
| 70.00th=[ 239], 80.00th=[ 279], 90.00th=[ 351], 95.00th=[ 443],
| 99.00th=[ 617], 99.50th=[ 651], 99.90th=[ 953], 99.95th=[ 4463],
| 99.99th=[ 4463]
bw ( KiB/s): min= 8192, max=32899, per=11.92%, avg=13352.78, stdev=6201.40, samples=1006
iops : min= 2, max= 8, avg= 3.26, stdev= 1.51, samples=1006
lat (usec) : 750=0.24%, 1000=0.06%
lat (msec) : 2=0.30%, 4=0.06%, 10=0.42%, 20=0.79%, 50=1.52%
lat (msec) : 100=7.89%, 250=61.59%, 500=24.45%, 750=2.31%, 1000=0.30%
lat (msec) : >=2000=0.06%
cpu : usr=0.06%, sys=0.33%, ctx=124924, majf=0, minf=302
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,1648,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=109MiB/s (115MB/s), 109MiB/s-109MiB/s (115MB/s-115MB/s), io=6592MiB (6912MB), run=60237-60237msec
Disk stats (read/write):
md3: ios=0/13185, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=3017/25338, aggrmerge=22222/137274, aggrticks=213182/525245, aggrin_queue=742103, aggrutil=72.74%
sdb: ios=3312/28322, merge=23796/149975, ticks=233614/539340, in_queue=776245, util=58.65%
sdh: ios=3337/28646, merge=23819/150204, ticks=217628/502871, in_queue=723089, util=54.61%
sdr: ios=3283/27886, merge=24161/150246, ticks=236231/551306, in_queue=790951, util=57.57%
sdw: ios=3234/27876, merge=23632/150093, ticks=197741/570927, in_queue=773071, util=56.04%
sdab: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
sdah: ios=3337/27699, merge=24931/148440, ticks=223557/528575, in_queue=755436, util=57.30%
sdaq: ios=3297/27670, merge=24071/150373, ticks=228849/514207, in_queue=746421, util=54.82%
sdar: ios=3413/27517, merge=24811/149363, ticks=214857/545679, in_queue=764002, util=56.38%
sdas: ios=3291/27569, merge=24200/149359, ticks=223544/564152, in_queue=792332, util=58.18%
sdat: ios=3385/27607, merge=24905/148955, ticks=236401/580202, in_queue=820073, util=57.14%
sdau: ios=3400/28016, merge=24908/149526, ticks=218847/539694, in_queue=762857, util=57.11%
sdav: ios=2925/25252, merge=23439/150758, ticks=326921/865997, in_queue=1200764, util=72.74%
```
* Testing with another array on `np04-srv-004` showed even better results: `1386MB/s` vs `64.8MB/s`! No significant CPU load was observed.
```shell=
[root@np04-srv-004 raid]# fio --name=seqwrite --rw=write --direct=1 --ioengine=libaio --bs=1024k --numjobs=10 --size=400G --runtime=6
0 --filename=/data1/bench_mark.trash
seqwrite: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
...
fio-3.1
Starting 10 processes
seqwrite: Laying out IO file (1 file / 409600MiB)
Jobs: 10 (f=10): [W(10)][100.0%][r=0KiB/s,w=1370MiB/s][r=0,w=1370 IOPS][eta 00m:00s]
seqwrite: (groupid=0, jobs=1): err= 0: pid=35336: Mon Oct 4 17:20:38 2021
write: IOPS=132, BW=132MiB/s (139MB/s)(7931MiB/60003msec)
slat (usec): min=81, max=140730, avg=234.25, stdev=3447.89
clat (usec): min=1334, max=54369, avg=7329.90, stdev=5236.82
lat (usec): min=1431, max=151114, avg=7564.31, stdev=6258.16
clat percentiles (usec):
| 1.00th=[ 1483], 5.00th=[ 1696], 10.00th=[ 1909], 20.00th=[ 2311],
| 30.00th=[ 2704], 40.00th=[ 5932], 50.00th=[ 7504], 60.00th=[ 8717],
| 70.00th=[ 9765], 80.00th=[10814], 90.00th=[12256], 95.00th=[13566],
| 99.00th=[30278], 99.50th=[37487], 99.90th=[46924], 99.95th=[49546],
| 99.99th=[54264]
bw ( KiB/s): min=90292, max=143647, per=10.01%, avg=135475.59, stdev=7350.31, samples=120
iops : min= 88, max= 140, avg=132.17, stdev= 7.18, samples=120
lat (msec) : 2=12.23%, 4=21.41%, 10=39.02%, 20=25.87%, 50=1.42%
lat (msec) : 100=0.04%
cpu : usr=0.26%, sys=1.17%, ctx=8000, majf=0, minf=30
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,7931,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
seqwrite: (groupid=0, jobs=1): err= 0: pid=35337: Mon Oct 4 17:20:38 2021
write: IOPS=132, BW=132MiB/s (139MB/s)(7935MiB/60001msec)
slat (usec): min=81, max=133372, avg=238.87, stdev=3570.96
clat (usec): min=1309, max=58295, avg=7321.10, stdev=5130.57
lat (usec): min=1406, max=142380, avg=7560.13, stdev=6260.20
clat percentiles (usec):
| 1.00th=[ 1483], 5.00th=[ 1680], 10.00th=[ 1909], 20.00th=[ 2311],
| 30.00th=[ 2704], 40.00th=[ 5932], 50.00th=[ 7504], 60.00th=[ 8717],
| 70.00th=[ 9765], 80.00th=[10945], 90.00th=[12256], 95.00th=[13566],
| 99.00th=[28181], 99.50th=[34866], 99.90th=[46400], 99.95th=[48497],
| 99.99th=[58459]
bw ( KiB/s): min=104448, max=143647, per=10.01%, avg=135573.27, stdev=7187.25, samples=120
iops : min= 102, max= 140, avg=132.23, stdev= 7.02, samples=120
lat (msec) : 2=12.33%, 4=21.42%, 10=38.88%, 20=25.86%, 50=1.49%
lat (msec) : 100=0.03%
cpu : usr=0.24%, sys=1.19%, ctx=8002, majf=0, minf=29
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,7935,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
seqwrite: (groupid=0, jobs=1): err= 0: pid=35338: Mon Oct 4 17:20:38 2021
write: IOPS=132, BW=132MiB/s (139MB/s)(7938MiB/60003msec)
slat (usec): min=67, max=128184, avg=212.83, stdev=3350.17
clat (usec): min=1296, max=61464, avg=7344.59, stdev=5205.38
lat (usec): min=1369, max=131783, avg=7557.59, stdev=6151.63
clat percentiles (usec):
| 1.00th=[ 1401], 5.00th=[ 1614], 10.00th=[ 1844], 20.00th=[ 2278],
| 30.00th=[ 2704], 40.00th=[ 5932], 50.00th=[ 7504], 60.00th=[ 8717],
| 70.00th=[ 9765], 80.00th=[10945], 90.00th=[12387], 95.00th=[13566],
| 99.00th=[29230], 99.50th=[35914], 99.90th=[45351], 99.95th=[51119],
| 99.99th=[61604]
bw ( KiB/s): min=108544, max=143360, per=10.01%, avg=135573.88, stdev=6587.45, samples=120
iops : min= 106, max= 140, avg=132.28, stdev= 6.42, samples=120
lat (msec) : 2=13.58%, 4=20.16%, 10=37.81%, 20=26.98%, 50=1.42%
lat (msec) : 100=0.05%
cpu : usr=0.26%, sys=0.93%, ctx=8009, majf=0, minf=28
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,7938,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
seqwrite: (groupid=0, jobs=1): err= 0: pid=35339: Mon Oct 4 17:20:38 2021
write: IOPS=132, BW=132MiB/s (139MB/s)(7933MiB/60002msec)
slat (usec): min=65, max=103747, avg=217.12, stdev=3314.65
clat (usec): min=1295, max=62116, avg=7344.97, stdev=5260.79
lat (usec): min=1403, max=112920, avg=7562.25, stdev=6247.53
clat percentiles (usec):
| 1.00th=[ 1401], 5.00th=[ 1631], 10.00th=[ 1844], 20.00th=[ 2278],
| 30.00th=[ 2704], 40.00th=[ 5932], 50.00th=[ 7570], 60.00th=[ 8717],
| 70.00th=[ 9765], 80.00th=[10945], 90.00th=[12387], 95.00th=[13566],
| 99.00th=[28181], 99.50th=[37487], 99.90th=[49021], 99.95th=[51643],
| 99.99th=[62129]
bw ( KiB/s): min=112640, max=143360, per=10.01%, avg=135489.83, stdev=6218.12, samples=120
iops : min= 110, max= 140, avg=132.20, stdev= 6.07, samples=120
lat (msec) : 2=13.32%, 4=20.51%, 10=38.12%, 20=26.55%, 50=1.42%
lat (msec) : 100=0.08%
cpu : usr=0.28%, sys=0.90%, ctx=7994, majf=0, minf=30
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,7933,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
seqwrite: (groupid=0, jobs=1): err= 0: pid=35340: Mon Oct 4 17:20:38 2021
write: IOPS=132, BW=132MiB/s (139MB/s)(7936MiB/60006msec)
slat (usec): min=65, max=115383, avg=213.45, stdev=3434.09
clat (usec): min=1296, max=50214, avg=7346.30, stdev=5146.90
lat (usec): min=1387, max=145000, avg=7559.90, stdev=6294.88
clat percentiles (usec):
| 1.00th=[ 1401], 5.00th=[ 1598], 10.00th=[ 1844], 20.00th=[ 2278],
| 30.00th=[ 2704], 40.00th=[ 5997], 50.00th=[ 7570], 60.00th=[ 8717],
| 70.00th=[ 9765], 80.00th=[10945], 90.00th=[12387], 95.00th=[13698],
| 99.00th=[29230], 99.50th=[34866], 99.90th=[45351], 99.95th=[46924],
| 99.99th=[50070]
bw ( KiB/s): min=100352, max=143360, per=10.01%, avg=135541.07, stdev=7368.16, samples=120
iops : min= 98, max= 140, avg=132.25, stdev= 7.19, samples=120
lat (msec) : 2=13.51%, 4=19.93%, 10=38.34%, 20=26.73%, 50=1.47%
lat (msec) : 100=0.01%
cpu : usr=0.27%, sys=0.89%, ctx=8025, majf=0, minf=30
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,7936,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
seqwrite: (groupid=0, jobs=1): err= 0: pid=35341: Mon Oct 4 17:20:38 2021
write: IOPS=132, BW=132MiB/s (139MB/s)(7935MiB/60004msec)
slat (usec): min=67, max=115435, avg=221.76, stdev=3388.66
clat (usec): min=1301, max=61395, avg=7338.69, stdev=5227.58
lat (usec): min=1398, max=117842, avg=7560.62, stdev=6222.18
clat percentiles (usec):
| 1.00th=[ 1401], 5.00th=[ 1598], 10.00th=[ 1844], 20.00th=[ 2278],
| 30.00th=[ 2671], 40.00th=[ 5932], 50.00th=[ 7570], 60.00th=[ 8717],
| 70.00th=[ 9765], 80.00th=[10945], 90.00th=[12387], 95.00th=[13566],
| 99.00th=[27657], 99.50th=[36439], 99.90th=[46400], 99.95th=[49021],
| 99.99th=[61604]
bw ( KiB/s): min=110813, max=141595, per=10.01%, avg=135506.31, stdev=6191.38, samples=120
iops : min= 108, max= 138, avg=132.22, stdev= 6.06, samples=120
lat (msec) : 2=13.76%, 4=19.95%, 10=38.00%, 20=26.87%, 50=1.39%
lat (msec) : 100=0.04%
cpu : usr=0.26%, sys=0.94%, ctx=8020, majf=0, minf=28
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,7935,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
seqwrite: (groupid=0, jobs=1): err= 0: pid=35342: Mon Oct 4 17:20:38 2021
write: IOPS=132, BW=132MiB/s (139MB/s)(7936MiB/60022msec)
slat (usec): min=67, max=107952, avg=228.76, stdev=3466.94
clat (usec): min=1264, max=53956, avg=7332.99, stdev=5119.19
lat (usec): min=1344, max=111823, avg=7561.92, stdev=6135.17
clat percentiles (usec):
| 1.00th=[ 1401], 5.00th=[ 1598], 10.00th=[ 1827], 20.00th=[ 2278],
| 30.00th=[ 2704], 40.00th=[ 6063], 50.00th=[ 7504], 60.00th=[ 8717],
| 70.00th=[ 9765], 80.00th=[10945], 90.00th=[12387], 95.00th=[13698],
| 99.00th=[27657], 99.50th=[34866], 99.90th=[43779], 99.95th=[46924],
| 99.99th=[53740]
bw ( KiB/s): min=110592, max=143647, per=10.01%, avg=135541.68, stdev=6354.25, samples=120
iops : min= 108, max= 140, avg=132.25, stdev= 6.19, samples=120
lat (msec) : 2=13.76%, 4=19.85%, 10=38.32%, 20=26.60%, 50=1.45%
lat (msec) : 100=0.03%
cpu : usr=0.25%, sys=0.94%, ctx=8011, majf=0, minf=29
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,7936,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
seqwrite: (groupid=0, jobs=1): err= 0: pid=35343: Mon Oct 4 17:20:38 2021
write: IOPS=132, BW=132MiB/s (139MB/s)(7934MiB/60007msec)
slat (usec): min=69, max=135236, avg=223.36, stdev=3572.92
clat (usec): min=1290, max=55551, avg=7338.35, stdev=5130.41
lat (usec): min=1372, max=137635, avg=7561.88, stdev=6281.55
clat percentiles (usec):
| 1.00th=[ 1401], 5.00th=[ 1614], 10.00th=[ 1844], 20.00th=[ 2278],
| 30.00th=[ 2704], 40.00th=[ 5997], 50.00th=[ 7570], 60.00th=[ 8717],
| 70.00th=[ 9765], 80.00th=[10814], 90.00th=[12387], 95.00th=[13566],
| 99.00th=[28443], 99.50th=[35390], 99.90th=[45876], 99.95th=[47449],
| 99.99th=[55313]
bw ( KiB/s): min=92344, max=143647, per=10.01%, avg=135506.65, stdev=7588.14, samples=120
iops : min= 90, max= 140, avg=132.22, stdev= 7.41, samples=120
lat (msec) : 2=13.61%, 4=19.57%, 10=38.66%, 20=26.67%, 50=1.46%
lat (msec) : 100=0.03%
cpu : usr=0.25%, sys=0.97%, ctx=8002, majf=0, minf=29
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,7934,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
seqwrite: (groupid=0, jobs=1): err= 0: pid=35344: Mon Oct 4 17:20:38 2021
write: IOPS=132, BW=132MiB/s (139MB/s)(7939MiB/60004msec)
slat (usec): min=70, max=102852, avg=220.50, stdev=3359.28
clat (usec): min=1271, max=55696, avg=7336.08, stdev=5153.16
lat (usec): min=1358, max=107574, avg=7556.73, stdev=6116.12
clat percentiles (usec):
| 1.00th=[ 1418], 5.00th=[ 1614], 10.00th=[ 1844], 20.00th=[ 2278],
| 30.00th=[ 2704], 40.00th=[ 5997], 50.00th=[ 7570], 60.00th=[ 8717],
| 70.00th=[ 9765], 80.00th=[10945], 90.00th=[12387], 95.00th=[13566],
| 99.00th=[27395], 99.50th=[35390], 99.90th=[46924], 99.95th=[50070],
| 99.99th=[55837]
bw ( KiB/s): min=116736, max=143360, per=10.02%, avg=135592.73, stdev=6222.06, samples=120
iops : min= 114, max= 140, avg=132.30, stdev= 6.07, samples=120
lat (msec) : 2=13.45%, 4=19.83%, 10=38.91%, 20=26.35%, 50=1.44%
lat (msec) : 100=0.03%
cpu : usr=0.25%, sys=1.01%, ctx=8009, majf=0, minf=26
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,7939,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
seqwrite: (groupid=0, jobs=1): err= 0: pid=35345: Mon Oct 4 17:20:38 2021
write: IOPS=132, BW=132MiB/s (139MB/s)(7934MiB/60005msec)
slat (usec): min=68, max=104473, avg=228.08, stdev=3391.34
clat (usec): min=1302, max=54880, avg=7333.49, stdev=5147.41
lat (usec): min=1392, max=117621, avg=7561.72, stdev=6158.26
clat percentiles (usec):
| 1.00th=[ 1401], 5.00th=[ 1614], 10.00th=[ 1860], 20.00th=[ 2278],
| 30.00th=[ 2737], 40.00th=[ 5997], 50.00th=[ 7570], 60.00th=[ 8717],
| 70.00th=[ 9765], 80.00th=[10814], 90.00th=[12387], 95.00th=[13566],
| 99.00th=[27395], 99.50th=[35914], 99.90th=[46924], 99.95th=[47973],
| 99.99th=[54789]
bw ( KiB/s): min=112640, max=143360, per=10.01%, avg=135506.81, stdev=6274.10, samples=120
iops : min= 110, max= 140, avg=132.22, stdev= 6.13, samples=120
lat (msec) : 2=13.08%, 4=20.33%, 10=38.69%, 20=26.43%, 50=1.42%
lat (msec) : 100=0.04%
cpu : usr=0.28%, sys=0.94%, ctx=8004, majf=0, minf=28
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,7934,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=1322MiB/s (1386MB/s), 132MiB/s-132MiB/s (139MB/s-139MB/s), io=77.5GiB (83.2GB), run=60001-60022msec
Disk stats (read/write):
md1: ios=0/158419, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/26586, aggrmerge=0/0, aggrticks=0/142500, aggrin_queue=142478, aggrutil=99.44%
sdk: ios=0/26587, merge=0/0, ticks=0/72957, in_queue=72928, util=83.90%
sdl: ios=0/26582, merge=0/0, ticks=0/67698, in_queue=67671, util=80.10%
sdm: ios=0/26582, merge=0/0, ticks=0/61810, in_queue=61787, util=75.16%
sdp: ios=0/26582, merge=0/0, ticks=0/62963, in_queue=62941, util=76.13%
sdq: ios=0/26582, merge=0/0, ticks=0/64183, in_queue=64168, util=77.92%
sdaa: ios=0/26591, merge=0/3, ticks=0/186935, in_queue=186908, util=98.80%
sdab: ios=0/26591, merge=0/3, ticks=0/190811, in_queue=190794, util=98.98%
sdac: ios=0/26590, merge=0/0, ticks=0/201569, in_queue=201550, util=98.93%
sdad: ios=0/26590, merge=0/0, ticks=0/197679, in_queue=197663, util=98.94%
sdae: ios=0/26586, merge=0/0, ticks=0/206380, in_queue=206349, util=99.42%
sdaf: ios=0/26586, merge=0/0, ticks=0/201825, in_queue=201800, util=99.44%
sdag: ios=0/26587, merge=0/0, ticks=0/195197, in_queue=195180, util=99.38%
[root@np04-srv-003 data3]# fio --name=seqwrite --rw=write --direct=1 --ioengine=libaio --bs=1024k --numjobs=10 --size=40
0G --runtime=60 --filename=/data1/bench_mark.trash
seqwrite: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
...
fio-3.1
Starting 10 processes
seqwrite: Laying out IO file (1 file / 409600MiB)
Jobs: 10 (f=10): [W(10)][100.0%][r=0KiB/s,w=46.0MiB/s][r=0,w=46 IOPS][eta 00m:00s]
seqwrite: (groupid=0, jobs=1): err= 0: pid=53093: Mon Oct 4 17:20:37 2021
write: IOPS=6, BW=6332KiB/s (6484kB/s)(373MiB/60323msec)
slat (usec): min=134, max=813239, avg=42991.57, stdev=110202.48
clat (msec): min=5, max=308, avg=118.72, stdev=65.32
lat (msec): min=6, max=972, avg=161.72, stdev=126.60
clat percentiles (msec):
| 1.00th=[ 9], 5.00th=[ 15], 10.00th=[ 37], 20.00th=[ 62],
| 30.00th=[ 78], 40.00th=[ 93], 50.00th=[ 116], 60.00th=[ 133],
| 70.00th=[ 150], 80.00th=[ 176], 90.00th=[ 209], 95.00th=[ 232],
| 99.00th=[ 288], 99.50th=[ 300], 99.90th=[ 309], 99.95th=[ 309],
| 99.99th=[ 309]
bw ( KiB/s): min= 2048, max=12312, per=10.30%, avg=6518.22, stdev=2341.30, samples=117
iops : min= 2, max= 12, avg= 6.36, stdev= 2.28, samples=117
lat (msec) : 10=1.07%, 20=5.90%, 50=7.51%, 100=28.69%, 250=53.62%
lat (msec) : 500=3.22%
cpu : usr=0.02%, sys=0.36%, ctx=15305, majf=0, minf=29
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,373,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
seqwrite: (groupid=0, jobs=1): err= 0: pid=53094: Mon Oct 4 17:20:37 2021
write: IOPS=6, BW=6341KiB/s (6493kB/s)(373MiB/60239msec)
slat (usec): min=141, max=594942, avg=44954.86, stdev=90161.61
clat (msec): min=11, max=391, avg=116.54, stdev=61.28
lat (msec): min=11, max=708, avg=161.49, stdev=104.91
clat percentiles (msec):
| 1.00th=[ 14], 5.00th=[ 23], 10.00th=[ 40], 20.00th=[ 66],
| 30.00th=[ 78], 40.00th=[ 100], 50.00th=[ 112], 60.00th=[ 130],
| 70.00th=[ 144], 80.00th=[ 165], 90.00th=[ 194], 95.00th=[ 220],
| 99.00th=[ 288], 99.50th=[ 334], 99.90th=[ 393], 99.95th=[ 393],
| 99.99th=[ 393]
bw ( KiB/s): min= 2052, max=10281, per=10.15%, avg=6422.28, stdev=2149.25, samples=119
iops : min= 2, max= 10, avg= 6.25, stdev= 2.09, samples=119
lat (msec) : 20=4.29%, 50=8.04%, 100=28.69%, 250=55.76%, 500=3.22%
cpu : usr=0.02%, sys=0.38%, ctx=16039, majf=0, minf=28
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,373,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
seqwrite: (groupid=0, jobs=1): err= 0: pid=53095: Mon Oct 4 17:20:37 2021
write: IOPS=6, BW=6327KiB/s (6478kB/s)(373MiB/60373msec)
slat (usec): min=137, max=693064, avg=43572.41, stdev=101652.63
clat (msec): min=7, max=308, avg=118.28, stdev=61.39
lat (msec): min=8, max=949, avg=161.85, stdev=115.61
clat percentiles (msec):
| 1.00th=[ 12], 5.00th=[ 22], 10.00th=[ 40], 20.00th=[ 64],
| 30.00th=[ 81], 40.00th=[ 101], 50.00th=[ 114], 60.00th=[ 130],
| 70.00th=[ 146], 80.00th=[ 174], 90.00th=[ 203], 95.00th=[ 230],
| 99.00th=[ 271], 99.50th=[ 279], 99.90th=[ 309], 99.95th=[ 309],
| 99.99th=[ 309]
bw ( KiB/s): min= 2048, max=12312, per=10.57%, avg=6690.04, stdev=2149.36, samples=114
iops : min= 2, max= 12, avg= 6.53, stdev= 2.10, samples=114
lat (msec) : 10=0.54%, 20=4.02%, 50=7.77%, 100=27.35%, 250=57.64%
lat (msec) : 500=2.68%
cpu : usr=0.03%, sys=0.30%, ctx=13513, majf=0, minf=27
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,373,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
seqwrite: (groupid=0, jobs=1): err= 0: pid=53096: Mon Oct 4 17:20:37 2021
write: IOPS=6, BW=6345KiB/s (6497kB/s)(373MiB/60199msec)
slat (usec): min=133, max=790693, avg=42577.58, stdev=108738.97
clat (msec): min=5, max=391, avg=118.81, stdev=65.00
lat (msec): min=6, max=866, avg=161.39, stdev=122.45
clat percentiles (msec):
| 1.00th=[ 12], 5.00th=[ 18], 10.00th=[ 40], 20.00th=[ 66],
| 30.00th=[ 83], 40.00th=[ 97], 50.00th=[ 109], 60.00th=[ 128],
| 70.00th=[ 148], 80.00th=[ 169], 90.00th=[ 213], 95.00th=[ 234],
| 99.00th=[ 326], 99.50th=[ 347], 99.90th=[ 393], 99.95th=[ 393],
| 99.99th=[ 393]
bw ( KiB/s): min= 2048, max=12312, per=10.30%, avg=6518.12, stdev=2432.09, samples=117
iops : min= 2, max= 12, avg= 6.36, stdev= 2.37, samples=117
lat (msec) : 10=0.80%, 20=5.36%, 50=5.90%, 100=30.03%, 250=54.69%
lat (msec) : 500=3.22%
cpu : usr=0.03%, sys=0.31%, ctx=14832, majf=0, minf=27
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,373,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
seqwrite: (groupid=0, jobs=1): err= 0: pid=53097: Mon Oct 4 17:20:37 2021
write: IOPS=6, BW=6353KiB/s (6505kB/s)(373MiB/60122msec)
slat (usec): min=139, max=767239, avg=44022.40, stdev=98919.49
clat (msec): min=8, max=348, avg=117.16, stdev=61.27
lat (msec): min=10, max=901, avg=161.18, stdev=115.83
clat percentiles (msec):
| 1.00th=[ 12], 5.00th=[ 22], 10.00th=[ 41], 20.00th=[ 66],
| 30.00th=[ 83], 40.00th=[ 99], 50.00th=[ 113], 60.00th=[ 129],
| 70.00th=[ 144], 80.00th=[ 161], 90.00th=[ 192], 95.00th=[ 224],
| 99.00th=[ 309], 99.50th=[ 334], 99.90th=[ 351], 99.95th=[ 351],
| 99.99th=[ 351]
bw ( KiB/s): min= 2048, max=10260, per=10.22%, avg=6462.92, stdev=2331.68, samples=118
iops : min= 2, max= 10, avg= 6.31, stdev= 2.27, samples=118
lat (msec) : 10=0.27%, 20=4.56%, 50=7.24%, 100=28.95%, 250=55.50%
lat (msec) : 500=3.49%
cpu : usr=0.04%, sys=0.31%, ctx=15451, majf=0, minf=28
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,373,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
seqwrite: (groupid=0, jobs=1): err= 0: pid=53098: Mon Oct 4 17:20:37 2021
write: IOPS=6, BW=6374KiB/s (6527kB/s)(374MiB/60086msec)
slat (usec): min=138, max=97080, avg=676.62, stdev=5002.99
clat (msec): min=12, max=457, avg=159.97, stdev=77.59
lat (msec): min=12, max=458, avg=160.65, stdev=77.45
clat percentiles (msec):
| 1.00th=[ 31], 5.00th=[ 60], 10.00th=[ 72], 20.00th=[ 94],
| 30.00th=[ 110], 40.00th=[ 131], 50.00th=[ 148], 60.00th=[ 169],
| 70.00th=[ 192], 80.00th=[ 222], 90.00th=[ 264], 95.00th=[ 305],
| 99.00th=[ 393], 99.50th=[ 397], 99.90th=[ 460], 99.95th=[ 460],
| 99.99th=[ 460]
bw ( KiB/s): min= 2048, max=10260, per=10.07%, avg=6372.50, stdev=1778.82, samples=120
iops : min= 2, max= 10, avg= 6.22, stdev= 1.74, samples=120
lat (msec) : 20=0.27%, 50=3.21%, 100=21.66%, 250=62.83%, 500=12.03%
cpu : usr=0.02%, sys=0.24%, ctx=398, majf=0, minf=28
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,374,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
seqwrite: (groupid=0, jobs=1): err= 0: pid=53099: Mon Oct 4 17:20:37 2021
write: IOPS=6, BW=6365KiB/s (6517kB/s)(373MiB/60012msec)
slat (usec): min=126, max=582570, avg=46946.39, stdev=93458.77
clat (msec): min=5, max=392, avg=113.94, stdev=67.55
lat (msec): min=6, max=730, avg=160.88, stdev=110.36
clat percentiles (msec):
| 1.00th=[ 9], 5.00th=[ 16], 10.00th=[ 21], 20.00th=[ 58],
| 30.00th=[ 75], 40.00th=[ 95], 50.00th=[ 109], 60.00th=[ 127],
| 70.00th=[ 144], 80.00th=[ 165], 90.00th=[ 197], 95.00th=[ 234],
| 99.00th=[ 330], 99.50th=[ 393], 99.90th=[ 393], 99.95th=[ 393],
| 99.99th=[ 393]
bw ( KiB/s): min= 2048, max=10260, per=10.13%, avg=6408.74, stdev=2225.83, samples=119
iops : min= 2, max= 10, avg= 6.25, stdev= 2.17, samples=119
lat (msec) : 10=1.61%, 20=8.04%, 50=6.97%, 100=27.35%, 250=52.28%
lat (msec) : 500=3.75%
cpu : usr=0.03%, sys=0.34%, ctx=16357, majf=0, minf=28
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,373,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
seqwrite: (groupid=0, jobs=1): err= 0: pid=53100: Mon Oct 4 17:20:37 2021
write: IOPS=6, BW=6357KiB/s (6509kB/s)(373MiB/60086msec)
slat (usec): min=133, max=677098, avg=36229.03, stdev=89448.04
clat (msec): min=6, max=421, avg=124.85, stdev=68.00
lat (msec): min=8, max=842, avg=161.08, stdev=106.48
clat percentiles (msec):
| 1.00th=[ 12], 5.00th=[ 25], 10.00th=[ 43], 20.00th=[ 68],
| 30.00th=[ 85], 40.00th=[ 102], 50.00th=[ 116], 60.00th=[ 134],
| 70.00th=[ 153], 80.00th=[ 178], 90.00th=[ 220], 95.00th=[ 249],
| 99.00th=[ 334], 99.50th=[ 393], 99.90th=[ 422], 99.95th=[ 422],
| 99.99th=[ 422]
bw ( KiB/s): min= 2048, max=10260, per=10.13%, avg=6408.91, stdev=2111.00, samples=119
iops : min= 2, max= 10, avg= 6.25, stdev= 2.06, samples=119
lat (msec) : 10=0.54%, 20=3.75%, 50=7.51%, 100=27.61%, 250=56.57%
lat (msec) : 500=4.02%
cpu : usr=0.05%, sys=0.29%, ctx=13348, majf=0, minf=28
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,373,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
seqwrite: (groupid=0, jobs=1): err= 0: pid=53101: Mon Oct 4 17:20:37 2021
write: IOPS=6, BW=6346KiB/s (6498kB/s)(372MiB/60028msec)
slat (usec): min=135, max=856278, avg=49038.92, stdev=101883.11
clat (msec): min=6, max=421, avg=112.32, stdev=63.71
lat (msec): min=10, max=983, avg=161.36, stdev=114.29
clat percentiles (msec):
| 1.00th=[ 9], 5.00th=[ 17], 10.00th=[ 33], 20.00th=[ 62],
| 30.00th=[ 77], 40.00th=[ 89], 50.00th=[ 105], 60.00th=[ 124],
| 70.00th=[ 142], 80.00th=[ 161], 90.00th=[ 194], 95.00th=[ 228],
| 99.00th=[ 288], 99.50th=[ 288], 99.90th=[ 422], 99.95th=[ 422],
| 99.99th=[ 422]
bw ( KiB/s): min= 2048, max=12312, per=10.19%, avg=6445.97, stdev=2198.76, samples=118
iops : min= 2, max= 12, avg= 6.29, stdev= 2.14, samples=118
lat (msec) : 10=1.34%, 20=6.72%, 50=7.53%, 100=31.72%, 250=49.19%
lat (msec) : 500=3.49%
cpu : usr=0.03%, sys=0.35%, ctx=16537, majf=0, minf=28
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,372,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
seqwrite: (groupid=0, jobs=1): err= 0: pid=53102: Mon Oct 4 17:20:37 2021
write: IOPS=6, BW=6327KiB/s (6478kB/s)(373MiB/60373msec)
slat (usec): min=139, max=795568, avg=44965.28, stdev=101384.60
clat (msec): min=7, max=339, avg=116.89, stdev=63.73
lat (msec): min=11, max=921, avg=161.85, stdev=117.48
clat percentiles (msec):
| 1.00th=[ 11], 5.00th=[ 17], 10.00th=[ 29], 20.00th=[ 61],
| 30.00th=[ 81], 40.00th=[ 96], 50.00th=[ 112], 60.00th=[ 130],
| 70.00th=[ 148], 80.00th=[ 169], 90.00th=[ 203], 95.00th=[ 226],
| 99.00th=[ 288], 99.50th=[ 309], 99.90th=[ 338], 99.95th=[ 338],
| 99.99th=[ 338]
bw ( KiB/s): min= 2048, max=12288, per=10.22%, avg=6462.95, stdev=2406.27, samples=118
iops : min= 2, max= 12, avg= 6.31, stdev= 2.35, samples=118
lat (msec) : 10=0.80%, 20=6.43%, 50=6.70%, 100=28.15%, 250=54.96%
lat (msec) : 500=2.95%
cpu : usr=0.03%, sys=0.32%, ctx=15103, majf=0, minf=28
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,373,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=61.8MiB/s (64.8MB/s), 6327KiB/s-6374KiB/s (6478kB/s-6527kB/s), io=3730MiB (3911MB), run=60012-60373msec
Disk stats (read/write):
md1: ios=0/7475, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=1075/21158, aggrmerge=6048/73265, aggrticks=38880/228852, aggrin_queue=267976, aggrutil=31.38%
sdo: ios=1140/23114, merge=6518/79208, ticks=41429/238697, in_queue=280372, util=26.90%
sdp: ios=1219/23066, merge=6615/80325, ticks=51421/233940, in_queue=285667, util=27.13%
sdq: ios=1184/23037, merge=6645/80383, ticks=43211/214274, in_queue=257724, util=26.39%
sds: ios=1101/22890, merge=6454/79678, ticks=38719/243882, in_queue=282886, util=28.45%
sdt: ios=1185/22721, merge=6538/79365, ticks=45132/322021, in_queue=367573, util=31.38%
sdu: ios=1233/22721, merge=6718/80298, ticks=42375/227935, in_queue=270579, util=27.07%
sdv: ios=1152/23294, merge=6683/81208, ticks=44656/279964, in_queue=324755, util=29.95%
sdx: ios=1183/23431, merge=6673/80284, ticks=37803/245268, in_queue=283253, util=28.58%
sdy: ios=1218/23349, merge=6637/80542, ticks=35094/226741, in_queue=261962, util=26.77%
sdz: ios=1162/23285, merge=6617/79672, ticks=41710/252131, in_queue=294101, util=28.36%
sdaa: ios=1125/22991, merge=6483/78224, ticks=45016/261381, in_queue=306844, util=28.96%
sdac: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
```
Sign in with Wallet
Connect another wallet