# Apollo 4200 Gen9 Data Fabric Performance
## 測試環境介紹
Apollo 4200 Gen9 *1
```
1* CPU E5-2620 v4 @ 2.10GHz
1* Memory 64G
3* RI SSD VK000960GWSRT
9* 7.2K HDD MB010000GWAYN
1* Mellanox Technologies MT27520 40G
```
Apollo 4200 Gen9 *3
```
1* CPU E5-2620 v4 @ 2.10GHz
1* Memory 64G
3* RI SSD VK000960GWSRT
9* 7.2K HDD MB010000GWAYN
1* Mellanox Technologies MT27520 40G
```
叢集架構
* 1 node DataFabric 6.2.0.0.20200915234957.GA
* Apollo 4200 Gen9 *1
* Storage pool 1 SSD *3 Label ssd
* Storage pool 2 HDD *3 Label default
* Storage pool 3 HDD *3 Label default
* Storage pool 4 HDD *3 Label default
* 3 node DataFabric 6.2.0.0.20200915234957.GA
* Apollo 4200 Gen9 *3
* Storage pool 1 SSD *3 Label ssd
* Storage pool 2 HDD *3 Label default
* Storage pool 3 HDD *3 Label default
* Storage pool 4 HDD *3 Label default
---
> Disk Layout
# Fio測試
## Fio測試參數說明
* -direct=1 > `不使用 io buffer`
* -iodepth 1 > `同一時間有多少 I/O 在做存取`
* -thread > `block size`
* -rw=randwrite > `測試隨機寫的I/O`
* -ioengine=psync > `io引擎使用pync方式`
* -bs=8k > `Block size`
* -size=1T > `檔案大小`
* -numjobs=100 > `模擬100個執行緒`
* -runtime=1000 > `測試時間為1000秒`
* -group_reporting > `關於顯示結果的，匯總每個執行緒的訊息。`
---
## 1 node ssd write
> fio -filename=/mapr/my.cluster.com/ssd/s1 -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=8k -size=1T -numjobs=100 -runtime=1000 -group_reporting -name=mytest
```shell=
mytest: (groupid=0, jobs=100): err= 0: pid=215407: Wed Jan 20 21:35:07 2021
write: IOPS=8522, BW=66.6MiB/s (69.8MB/s)(65.0GiB/1000012msec)
clat (usec): min=673, max=236227, avg=11717.86, stdev=4771.42
lat (usec): min=675, max=236228, avg=11718.54, stdev=4771.42
clat percentiles (msec):
| 1.00th=[ 7], 5.00th=[ 9], 10.00th=[ 9], 20.00th=[ 10],
| 30.00th=[ 11], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 12],
| 70.00th=[ 13], 80.00th=[ 14], 90.00th=[ 15], 95.00th=[ 16],
| 99.00th=[ 22], 99.50th=[ 34], 99.90th=[ 82], 99.95th=[ 103],
| 99.99th=[ 146]
bw ( KiB/s): min= 15, max= 1024, per=1.00%, avg=682.13, stdev=62.09, samples=199785
iops : min= 1, max= 128, avg=85.22, stdev= 7.77, samples=199785
lat (usec) : 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.07%, 10=23.36%, 20=75.40%, 50=0.89%
lat (msec) : 100=0.23%, 250=0.05%
cpu : usr=0.08%, sys=0.15%, ctx=8523219, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,8522466,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=66.6MiB/s (69.8MB/s), 66.6MiB/s-66.6MiB/s (69.8MB/s-69.8MB/s), io=65.0GiB (69.8GB), run=1000012-1000012msec
```
## 1 node ssd read
> fio -filename=/mapr/my.cluster.com/ssd/s1 -direct=1 -iodepth 1 -thread -rw=randread -ioengine=psync -bs=8k -size=1T -numjobs=100 -runtime=1000 -group_reporting -name=mytest
```shell=
mytest: (g=0): rw=randread, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=psync, iodepth=1
...shell=
fio-3.7
Starting 100 threads
mytest: Laying out IO file (1 file / 1048576MiB)
fio: pid=0, err=5/file:filesetup.c:223, func=write, error=Input/output error
Jobs: 99 (f=99): [X(1),r(99)][100.0%][r=117MiB/s,w=0KiB/s][r=14.9k,w=0 IOPS][eta 00m:00s]
mytest: (groupid=0, jobs=100): err= 5 (file:filesetup.c:223, func=write, error=Input/output error): pid=0: Wed Jan 20 22:03:58 2021
read: IOPS=13.6k, BW=106MiB/s (112MB/s)(104GiB/1000013msec)
clat (usec): min=103, max=79433, avg=6440.48, stdev=2651.28
lat (usec): min=103, max=79433, avg=6440.86, stdev=2651.28
clat percentiles (usec):
| 1.00th=[ 668], 5.00th=[ 2311], 10.00th=[ 3720], 20.00th=[ 4686],
| 30.00th=[ 5211], 40.00th=[ 5669], 50.00th=[ 6128], 60.00th=[ 6587],
| 70.00th=[ 7111], 80.00th=[ 7898], 90.00th=[ 9896], 95.00th=[11731],
| 99.00th=[14615], 99.50th=[15664], 99.90th=[17695], 99.95th=[19268],
| 99.99th=[23725]
bw ( KiB/s): min= 64, max= 2304, per=1.14%, avg=1240.38, stdev=98.82, samples=175476
iops : min= 8, max= 288, avg=155.01, stdev=12.36, samples=175476
lat (usec) : 250=0.04%, 500=0.52%, 750=0.64%, 1000=0.57%
lat (msec) : 2=2.38%, 4=7.78%, 10=78.33%, 20=9.70%, 50=0.04%
lat (msec) : 100=0.01%
cpu : usr=0.10%, sys=1.91%, ctx=16528132, majf=0, minf=198
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=13617345,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=106MiB/s (112MB/s), 106MiB/s-106MiB/s (112MB/s-112MB/s), io=104GiB (112GB), run=1000013-1000013msec
```
## 1 node hdd write
> fio -filename=/mapr/my.cluster.com/hdd/h1 -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=8k -size=1T -numjobs=100 -runtime=1000 -group_reporting -name=mytest
```shell=
Jobs: 100 (f=100): [w(100)][100.0%][r=0KiB/s,w=62.1MiB/s][r=0,w=7945 IOPS][eta 00m:00s]
mytest: (groupid=0, jobs=100): err= 0: pid=602876: Wed Jan 20 22:23:02 2021
write: IOPS=7845, BW=61.3MiB/s (64.3MB/s)(59.9GiB/1000016msec)
clat (usec): min=455, max=1842.2k, avg=12664.74, stdev=21455.76
lat (usec): min=456, max=1842.2k, avg=12665.47, stdev=21455.77
clat percentiles (usec):
| 1.00th=[ 1516], 5.00th=[ 4555], 10.00th=[ 6259], 20.00th=[ 7635],
| 30.00th=[ 8455], 40.00th=[ 9110], 50.00th=[ 9765], 60.00th=[ 10421],
| 70.00th=[ 11207], 80.00th=[ 12256], 90.00th=[ 14222], 95.00th=[ 18220],
| 99.00th=[103285], 99.50th=[162530], 99.90th=[267387], 99.95th=[337642],
| 99.99th=[566232]
bw ( KiB/s): min= 15, max= 1936, per=1.01%, avg=632.65, stdev=176.98, samples=198286
iops : min= 1, max= 242, avg=79.03, stdev=22.13, samples=198286
lat (usec) : 500=0.01%, 750=0.02%, 1000=0.32%
lat (msec) : 2=1.21%, 4=2.50%, 10=49.40%, 20=41.95%, 50=2.35%
lat (msec) : 100=1.21%, 250=0.91%, 500=0.11%, 750=0.01%, 1000=0.01%
cpu : usr=0.07%, sys=0.14%, ctx=7846243, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,7845578,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=61.3MiB/s (64.3MB/s), 61.3MiB/s-61.3MiB/s (64.3MB/s-64.3MB/s), io=59.9GiB (64.3GB), run=1000016-1000016msec
```
## 1 node hdd read
> fio -filename=/mapr/my.cluster.com/hdd/h1 -direct=1 -iodepth 1 -thread -rw=randread -ioengine=psync -bs=8k -size=1T -numjobs=100 -runtime=1000 -group_reporting -name=mytest
```shell=
fio-3.7
Starting 100 threads
mytest: Laying out IO file (1 file / 1048576MiB)
fio: pid=0, err=5/file:filesetup.c:223, func=write, error=Input/output error
Jobs: 31 (f=31): [X(1),_(3),r(1),_(1),r(1),_(4),r(1),E(1),r(1),_(2),r(1),_(8),r(1),_(3),r(1),_(4),r(1),_(1),r(1),_(3),r(1),_(7),r(1),_(1),r(1),_(3),r(1),_(6),r(1),_(1),r(6),_(1),r(1),_(2),r(1),_(5),r(2),_(2),r(1),_(3),r(3),_(3),r(1),_(3),r(1),_(1),r(1)][1.5%][r=114MiB/s,w=0KiB/s][r=14.5k,w=0 IOPS][eta 18h:36m:40s]
mytest: (groupid=0, jobs=100): err= 5 (file:filesetup.c:223, func=write, error=Input/output error): pid=0: Wed Jan 20 22:48:27 2021
read: IOPS=14.0k, BW=110MiB/s (115MB/s)(107GiB/1000056msec)
clat (usec): min=101, max=258415, avg=6560.39, stdev=5606.22
lat (usec): min=102, max=258416, avg=6560.76, stdev=5606.22
clat percentiles (usec):
| 1.00th=[ 619], 5.00th=[ 2147], 10.00th=[ 3097], 20.00th=[ 3949],
| 30.00th=[ 4490], 40.00th=[ 4948], 50.00th=[ 5342], 60.00th=[ 5735],
| 70.00th=[ 6194], 80.00th=[ 6915], 90.00th=[ 9634], 95.00th=[18482],
| 99.00th=[30016], 99.50th=[36439], 99.90th=[54264], 99.95th=[62653],
| 99.99th=[84411]
bw ( KiB/s): min= 16, max= 3072, per=1.09%, avg=1217.79, stdev=154.14, samples=184134
iops : min= 2, max= 384, avg=152.18, stdev=19.27, samples=184134
lat (usec) : 250=0.05%, 500=0.61%, 750=0.71%, 1000=0.64%
lat (msec) : 2=2.56%, 4=16.31%, 10=69.30%, 20=5.80%, 50=3.87%
lat (msec) : 100=0.14%, 250=0.01%, 500=0.01%
cpu : usr=0.10%, sys=1.26%, ctx=15752693, majf=0, minf=198
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=14028071,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=110MiB/s (115MB/s), 110MiB/s-110MiB/s (115MB/s-115MB/s), io=107GiB (115GB), run=1000056-1000056msec
```
## 3 node ssd write
> fio -filename=/mapr/my.cluster.com/ssd/s1 -direct=1 -iodepth 1 -thread -rw=randread -ioengine=psync -bs=8k -size=1T -numjobs=100 -runtime=1000 -group_reporting -name=mytest
```shell=
mytest: (groupid=0, jobs=100): err= 0: pid=883514: Thu Jan 21 21:52:11 2021
write: IOPS=7467, BW=58.3MiB/s (61.2MB/s)(56.0GiB/1000013msec)
clat (usec): min=1879, max=179728, avg=13353.26, stdev=3723.75
lat (usec): min=1879, max=179728, avg=13353.94, stdev=3723.75
clat percentiles (usec):
| 1.00th=[ 7701], 5.00th=[ 8979], 10.00th=[ 9634], 20.00th=[10683],
| 30.00th=[11469], 40.00th=[12256], 50.00th=[12911], 60.00th=[13698],
| 70.00th=[14615], 80.00th=[15664], 90.00th=[17171], 95.00th=[18744],
| 99.00th=[21890], 99.50th=[23725], 99.90th=[53740], 99.95th=[74974],
| 99.99th=[92799]
bw ( KiB/s): min= 16, max= 752, per=1.00%, avg=598.64, stdev=56.56, samples=199485
iops : min= 2, max= 94, avg=74.78, stdev= 7.08, samples=199485
lat (msec) : 2=0.01%, 4=0.01%, 10=12.78%, 20=84.65%, 50=2.46%
lat (msec) : 100=0.11%, 250=0.01%
cpu : usr=0.07%, sys=0.14%, ctx=7468243, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,7467606,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=58.3MiB/s (61.2MB/s), 58.3MiB/s-58.3MiB/s (61.2MB/s-61.2MB/s), io=56.0GiB (61.2GB), run=1000013-1000013msec
```
## 3 node ssd read
> fio -filename=/mapr/my.cluster.com/ssd/s1 -direct=1 -iodepth 1 -thread -rw=randread -ioengine=psync -bs=8k -size=1T -numjobs=100 -runtime=1000 -group_reporting -name=mytest
```shell=
mytest: Laying out IO file (1 file / 1048576MiB)
fio: pid=0, err=5/file:filesetup.c:223, func=write, error=Input/output error
Jobs: 99 (f=99): [X(1),r(99)][100.0%][r=130MiB/s,w=0KiB/s][r=16.6k,w=0 IOPS][eta 00m:00s]
mytest: (groupid=0, jobs=100): err= 5 (file:filesetup.c:223, func=write, error=Input/output error): pid=0: Thu Jan 21 22:22:59 2021
read: IOPS=15.4k, BW=120MiB/s (126MB/s)(117GiB/1000007msec)
clat (usec): min=95, max=35775, avg=5933.30, stdev=2321.72
lat (usec): min=95, max=35775, avg=5933.62, stdev=2321.72
clat percentiles (usec):
| 1.00th=[ 693], 5.00th=[ 2114], 10.00th=[ 3556], 20.00th=[ 4490],
| 30.00th=[ 4948], 40.00th=[ 5276], 50.00th=[ 5669], 60.00th=[ 6194],
| 70.00th=[ 6652], 80.00th=[ 7308], 90.00th=[ 8455], 95.00th=[10159],
| 99.00th=[13566], 99.50th=[14615], 99.90th=[16909], 99.95th=[17957],
| 99.99th=[21103]
bw ( KiB/s): min= 16, max= 2704, per=1.09%, avg=1346.44, stdev=130.07, samples=182821
iops : min= 2, max= 338, avg=168.27, stdev=16.27, samples=182821
lat (usec) : 100=0.01%, 250=0.04%, 500=0.47%, 750=0.66%, 1000=0.65%
lat (msec) : 2=2.85%, 4=8.56%, 10=81.36%, 20=5.41%, 50=0.02%
cpu : usr=0.11%, sys=1.44%, ctx=15802303, majf=0, minf=198
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=15400157,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=120MiB/s (126MB/s), 120MiB/s-120MiB/s (126MB/s-126MB/s), io=117GiB (126GB), run=1000007-1000007msec
```
## 3 node hdd write
> fio -filename=/mapr/my.cluster.com/hdd/h1 -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=8k -size=1T -numjobs=100 -runtime=1000 -group_reporting -name=mytest
```shell=
(groupid=0, jobs=100): err= 0: pid=1248396: Thu Jan 21 22:46:13 2021
write: IOPS=6176, BW=48.3MiB/s (50.6MB/s)(47.1GiB/1000087msec)
clat (usec): min=1822, max=2388.4k, avg=16183.54, stdev=32616.01
lat (usec): min=1823, max=2388.4k, avg=16184.24, stdev=32616.01
clat percentiles (msec):
| 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8],
| 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 13],
| 70.00th=[ 14], 80.00th=[ 16], 90.00th=[ 22], 95.00th=[ 40],
| 99.00th=[ 127], 99.50th=[ 186], 99.90th=[ 409], 99.95th=[ 693],
| 99.99th=[ 1133]
bw ( KiB/s): min= 15, max= 1184, per=1.02%, avg=502.25, stdev=165.05, samples=196684
iops : min= 1, max= 148, avg=62.73, stdev=20.64, samples=196684
lat (msec) : 2=0.01%, 4=0.95%, 10=37.63%, 20=50.53%, 50=7.34%
lat (msec) : 100=2.17%, 250=1.17%, 500=0.13%, 750=0.04%, 1000=0.02%
cpu : usr=0.06%, sys=0.12%, ctx=6177528, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,6177085,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=48.3MiB/s (50.6MB/s), 48.3MiB/s-48.3MiB/s (50.6MB/s-50.6MB/s), io=47.1GiB (50.6GB), run=1000087-1000087msec
```
## 3 node hdd read
> fio -filename=/mapr/my.cluster.com/hdd/h1 -direct=1 -iodepth 1 -thread -rw=randread -ioengine=psync -bs=8k -size=1T -numjobs=100 -runtime=1000 -group_reporting -name=mytest
```shell=
...
fio-3.7
Starting 100 threads
mytest: Laying out IO file (1 file / 1048576MiB)
fio: pid=0, err=5/file:filesetup.c:223, func=write, error=Input/output error
Jobs: 99 (f=99): [X(1),r(99)][100.0%][r=127MiB/s,w=0KiB/s][r=16.3k,w=0 IOPS][eta 00m:00s]
mytest: (groupid=0, jobs=100): err= 5 (file:filesetup.c:223, func=write, error=Input/output error): pid=0: Thu Jan 21 23:15:06 2021
read: IOPS=15.1k, BW=118MiB/s (124MB/s)(115GiB/1000041msec)
clat (usec): min=110, max=286385, avg=6092.74, stdev=5479.46
lat (usec): min=111, max=286385, avg=6093.07, stdev=5479.46
clat percentiles (usec):
| 1.00th=[ 668], 5.00th=[ 2089], 10.00th=[ 3097], 20.00th=[ 3687],
| 30.00th=[ 4015], 40.00th=[ 4359], 50.00th=[ 4686], 60.00th=[ 5080],
| 70.00th=[ 5669], 80.00th=[ 6390], 90.00th=[ 9503], 95.00th=[17433],
| 99.00th=[29492], 99.50th=[35914], 99.90th=[54264], 99.95th=[63177],
| 99.99th=[86508]
bw ( KiB/s): min= 16, max= 2528, per=1.09%, avg=1311.39, stdev=174.91, samples=183930
iops : min= 2, max= 316, avg=163.88, stdev=21.87, samples=183930
lat (usec) : 250=0.04%, 500=0.51%, 750=0.68%, 1000=0.64%
lat (msec) : 2=2.85%, 4=24.51%, 10=61.06%, 20=6.32%, 50=3.24%
lat (msec) : 100=0.14%, 250=0.01%, 500=0.01%
cpu : usr=0.10%, sys=1.35%, ctx=15451057, majf=0, minf=198
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=15088089,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=118MiB/s (124MB/s), 118MiB/s-118MiB/s (124MB/s-124MB/s), io=115GiB (124GB), run=1000041-1000041msec
```
### 4node hdd write Block=>8k
> fio -filename=/mapr/my.cluster.com/hdd/h1 -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=8k -size=1T -numjobs=100 -runtime=1000 -group_reporting -name=mytest
```shell=
mytest: (groupid=0, jobs=100): err= 0: pid=1560194: Tue Jan 26 21:41:54 2021
write: IOPS=4234, BW=33.1MiB/s (34.7MB/s)(32.3GiB/1000707msec)
clat (usec): min=1677, max=4705.1k, avg=23589.82, stdev=75792.02
lat (usec): min=1677, max=4705.1k, avg=23590.57, stdev=75792.02
clat percentiles (msec):
| 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 5],
| 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 10], 60.00th=[ 13],
| 70.00th=[ 16], 80.00th=[ 21], 90.00th=[ 31], 95.00th=[ 62],
| 99.00th=[ 330], 99.50th=[ 489], 99.90th=[ 1045], 99.95th=[ 1368],
| 99.99th=[ 1921]
bw ( KiB/s): min= 15, max= 1776, per=1.08%, avg=364.31, stdev=194.48, samples=186047
iops : min= 1, max= 222, avg=45.49, stdev=24.31, samples=186047
lat (msec) : 2=0.02%, 4=9.49%, 10=42.85%, 20=27.26%, 50=14.62%
lat (msec) : 100=2.08%, 250=2.17%, 500=1.02%, 750=0.27%, 1000=0.10%
cpu : usr=0.05%, sys=0.09%, ctx=4238538, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4237994,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=33.1MiB/s (34.7MB/s), 33.1MiB/s-33.1MiB/s (34.7MB/s-34.7MB/s), io=32.3GiB (34.7GB), run=1000707-1000707msec
```
### 4node hdd write Block=>256k
> fio -filename=/mapr/my.cluster.com/hdd/h1 -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=256k -size=1T -numjobs=100 -runtime=1000 -group_reporting -name=mytest
```shell=
mytest: (groupid=0, jobs=100): err= 0: pid=1482875: Tue Jan 26 21:24:06 2021
write: IOPS=1938, BW=485MiB/s (508MB/s)(473GiB/1000098msec)
clat (msec): min=2, max=3720, avg=51.57, stdev=64.74
lat (msec): min=2, max=3720, avg=51.58, stdev=64.74
clat percentiles (msec):
| 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 7], 20.00th=[ 10],
| 30.00th=[ 15], 40.00th=[ 49], 50.00th=[ 64], 60.00th=[ 68],
| 70.00th=[ 71], 80.00th=[ 75], 90.00th=[ 81], 95.00th=[ 87],
| 99.00th=[ 120], 99.50th=[ 251], 99.90th=[ 1099], 99.95th=[ 1351],
| 99.99th=[ 1653]
bw ( KiB/s): min= 509, max=13796, per=1.03%, avg=5106.18, stdev=1483.71, samples=194294
iops : min= 1, max= 53, avg=19.86, stdev= 5.80, samples=194294
lat (msec) : 4=0.02%, 10=22.47%, 20=12.41%, 50=5.26%, 100=58.14%
lat (msec) : 250=1.20%, 500=0.20%, 750=0.10%, 1000=0.07%
cpu : usr=0.05%, sys=0.13%, ctx=1938977, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1938405,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=485MiB/s (508MB/s), 485MiB/s-485MiB/s (508MB/s-508MB/s), io=473GiB (508GB), run=1000098-1000098msec
```
### 4node hdd write Block=>8k iodepth=>4
> fio -filename=/mapr/my.cluster.com/hdd/h1 -direct=4 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=8k -size=1T -numjobs=100 -runtime=1000 -group_reporting -name=mytest
```shell=
mytest: (groupid=0, jobs=100): err= 0: pid=1702829: Tue Jan 26 22:14:22 2021
write: IOPS=1652, BW=413MiB/s (433MB/s)(403GiB/1000063msec)
clat (msec): min=2, max=8775, avg=60.50, stdev=187.17
lat (msec): min=2, max=8775, avg=60.51, stdev=187.17
clat percentiles (msec):
| 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 11],
| 30.00th=[ 16], 40.00th=[ 25], 50.00th=[ 45], 60.00th=[ 65],
| 70.00th=[ 72], 80.00th=[ 79], 90.00th=[ 86], 95.00th=[ 93],
| 99.00th=[ 368], 99.50th=[ 995], 99.90th=[ 3239], 99.95th=[ 3809],
| 99.99th=[ 4597]
bw ( KiB/s): min= 507, max=20992, per=1.20%, avg=5061.16, stdev=1957.05, samples=167103
iops : min= 1, max= 82, avg=19.69, stdev= 7.65, samples=167103
lat (msec) : 4=0.09%, 10=19.30%, 20=16.87%, 50=15.95%, 100=44.64%
lat (msec) : 250=1.85%, 500=0.48%, 750=0.21%, 1000=0.11%
cpu : usr=0.04%, sys=0.11%, ctx=1652917, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1652376,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
WRITE: bw=413MiB/s (433MB/s), 413MiB/s-413MiB/s (433MB/s-433MB/s), io=403GiB (433GB), run=1000063-1000063msec
```
# dd 測試
## 3 node `dd` hdd write
```shell=
[root@node1 ~]# time dd if=/dev/zero of=/mapr/my.cluster.com/hdd/h1 bs=8k count=1024000
1024000+0 records in
1024000+0 records out
8388608000 bytes (8.4 GB, 7.8 GiB) copied, 8.41648 s, 997 MB/s
real 0m8.605s
user 0m1.102s
sys 0m6.686s
```
## 3 node `dd` hdd read
```shell=
[root@node1 ~]# time dd if=/mapr/my.cluster.com/hdd/h1 of=/dev/null bs=8k count=1024000
1024000+0 records in
1024000+0 records out
8388608000 bytes (8.4 GB, 7.8 GiB) copied, 8.75784 s, 958 MB/s
real 0m8.763s
user 0m1.594s
sys 0m6.919s
```
## 3 node `dd` ssd write
```shell=
[root@node1 ~]# time dd if=/dev/zero of=/mapr/my.cluster.com/ssd/s1 bs=8k count=1024000
1024000+0 records in
1024000+0 records out
8388608000 bytes (8.4 GB, 7.8 GiB) copied, 8.80219 s, 953 MB/s
real 0m8.828s
user 0m1.020s
sys 0m6.626s
```
## 3 node `dd` ssd read
```shell=
[root@node1 ~]# time dd if=/mapr/my.cluster.com/ssd/s1 of=/dev/null bs=8k count=1024000
1024000+0 records in
1024000+0 records out
8388608000 bytes (8.4 GB, 7.8 GiB) copied, 8.84118 s, 949 MB/s
real 0m8.846s
user 0m1.591s
sys 0m6.974s
```
Sign in with Wallet
Connect another wallet