---
# System prepended metadata

title: Apollo 4200 Gen9 Data Fabric Performance

---

# Apollo 4200 Gen9 Data Fabric Performance

## 測試環境介紹


Apollo 4200 Gen9 *1
```
1* CPU E5-2620 v4 @ 2.10GHz 
1* Memory 64G 
3* RI   SSD  VK000960GWSRT
9* 7.2K HDD  MB010000GWAYN
1* Mellanox Technologies MT27520 40G
```

Apollo 4200 Gen9 *3
```
1* CPU E5-2620 v4 @ 2.10GHz 
1* Memory 64G 
3* RI   SSD  VK000960GWSRT
9* 7.2K HDD  MB010000GWAYN
1* Mellanox Technologies MT27520 40G
```

叢集架構
* 1 node DataFabric 6.2.0.0.20200915234957.GA
    * Apollo 4200 Gen9 *1
        * Storage pool 1 SSD *3 Label ssd
        * Storage pool 2 HDD *3 Label default
        * Storage pool 3 HDD *3 Label default
        * Storage pool 4 HDD *3 Label default

* 3 node DataFabric 6.2.0.0.20200915234957.GA
    * Apollo 4200 Gen9 *3
        * Storage pool 1 SSD *3 Label ssd
        * Storage pool 2 HDD *3 Label default
        * Storage pool 3 HDD *3 Label default
        * Storage pool 4 HDD *3 Label default
---
> Disk Layout
![](https://i.imgur.com/H0oV1Vj.png)
![](https://i.imgur.com/69z3s2D.png)

# Fio測試

## Fio測試參數說明
* -direct=1 > `不使用 io buffer`
* -iodepth 1 > `同一時間有多少 I/O 在做存取`
* -thread > `block size`
* -rw=randwrite > `測試隨機寫的I/O`
* -ioengine=psync >  `io引擎使用pync方式`
* -bs=8k > `Block size`
* -size=1T > `檔案大小`
* -numjobs=100 > `模擬100個執行緒`
* -runtime=1000 > `測試時間為1000秒`
* -group_reporting > `關於顯示結果的，匯總每個執行緒的訊息。`



---

## 1 node ssd write
> fio -filename=/mapr/my.cluster.com/ssd/s1 -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=8k -size=1T -numjobs=100 -runtime=1000 -group_reporting -name=mytest
```shell=
mytest: (groupid=0, jobs=100): err= 0: pid=215407: Wed Jan 20 21:35:07 2021
  write: IOPS=8522, BW=66.6MiB/s (69.8MB/s)(65.0GiB/1000012msec)
    clat (usec): min=673, max=236227, avg=11717.86, stdev=4771.42
     lat (usec): min=675, max=236228, avg=11718.54, stdev=4771.42
    clat percentiles (msec):
     |  1.00th=[    7],  5.00th=[    9], 10.00th=[    9], 20.00th=[   10],
     | 30.00th=[   11], 40.00th=[   11], 50.00th=[   12], 60.00th=[   12],
     | 70.00th=[   13], 80.00th=[   14], 90.00th=[   15], 95.00th=[   16],
     | 99.00th=[   22], 99.50th=[   34], 99.90th=[   82], 99.95th=[  103],
     | 99.99th=[  146]
   bw (  KiB/s): min=   15, max= 1024, per=1.00%, avg=682.13, stdev=62.09, samples=199785
   iops        : min=    1, max=  128, avg=85.22, stdev= 7.77, samples=199785
  lat (usec)   : 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.07%, 10=23.36%, 20=75.40%, 50=0.89%
  lat (msec)   : 100=0.23%, 250=0.05%
  cpu          : usr=0.08%, sys=0.15%, ctx=8523219, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,8522466,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=66.6MiB/s (69.8MB/s), 66.6MiB/s-66.6MiB/s (69.8MB/s-69.8MB/s), io=65.0GiB (69.8GB), run=1000012-1000012msec
```

## 1 node ssd read
> fio -filename=/mapr/my.cluster.com/ssd/s1 -direct=1 -iodepth 1 -thread -rw=randread -ioengine=psync -bs=8k -size=1T -numjobs=100 -runtime=1000 -group_reporting -name=mytest
```shell=
mytest: (g=0): rw=randread, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=psync, iodepth=1
...shell=
fio-3.7
Starting 100 threads
mytest: Laying out IO file (1 file / 1048576MiB)
fio: pid=0, err=5/file:filesetup.c:223, func=write, error=Input/output error
Jobs: 99 (f=99): [X(1),r(99)][100.0%][r=117MiB/s,w=0KiB/s][r=14.9k,w=0 IOPS][eta 00m:00s]
mytest: (groupid=0, jobs=100): err= 5 (file:filesetup.c:223, func=write, error=Input/output error): pid=0: Wed Jan 20 22:03:58 2021
   read: IOPS=13.6k, BW=106MiB/s (112MB/s)(104GiB/1000013msec)
    clat (usec): min=103, max=79433, avg=6440.48, stdev=2651.28
     lat (usec): min=103, max=79433, avg=6440.86, stdev=2651.28
    clat percentiles (usec):
     |  1.00th=[  668],  5.00th=[ 2311], 10.00th=[ 3720], 20.00th=[ 4686],
     | 30.00th=[ 5211], 40.00th=[ 5669], 50.00th=[ 6128], 60.00th=[ 6587],
     | 70.00th=[ 7111], 80.00th=[ 7898], 90.00th=[ 9896], 95.00th=[11731],
     | 99.00th=[14615], 99.50th=[15664], 99.90th=[17695], 99.95th=[19268],
     | 99.99th=[23725]
   bw (  KiB/s): min=   64, max= 2304, per=1.14%, avg=1240.38, stdev=98.82, samples=175476
   iops        : min=    8, max=  288, avg=155.01, stdev=12.36, samples=175476
  lat (usec)   : 250=0.04%, 500=0.52%, 750=0.64%, 1000=0.57%
  lat (msec)   : 2=2.38%, 4=7.78%, 10=78.33%, 20=9.70%, 50=0.04%
  lat (msec)   : 100=0.01%
  cpu          : usr=0.10%, sys=1.91%, ctx=16528132, majf=0, minf=198
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=13617345,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=106MiB/s (112MB/s), 106MiB/s-106MiB/s (112MB/s-112MB/s), io=104GiB (112GB), run=1000013-1000013msec
```


## 1 node hdd write
> fio -filename=/mapr/my.cluster.com/hdd/h1 -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=8k -size=1T -numjobs=100 -runtime=1000 -group_reporting -name=mytest
```shell=
Jobs: 100 (f=100): [w(100)][100.0%][r=0KiB/s,w=62.1MiB/s][r=0,w=7945 IOPS][eta 00m:00s]
mytest: (groupid=0, jobs=100): err= 0: pid=602876: Wed Jan 20 22:23:02 2021
  write: IOPS=7845, BW=61.3MiB/s (64.3MB/s)(59.9GiB/1000016msec)
    clat (usec): min=455, max=1842.2k, avg=12664.74, stdev=21455.76
     lat (usec): min=456, max=1842.2k, avg=12665.47, stdev=21455.77
    clat percentiles (usec):
     |  1.00th=[  1516],  5.00th=[  4555], 10.00th=[  6259], 20.00th=[  7635],
     | 30.00th=[  8455], 40.00th=[  9110], 50.00th=[  9765], 60.00th=[ 10421],
     | 70.00th=[ 11207], 80.00th=[ 12256], 90.00th=[ 14222], 95.00th=[ 18220],
     | 99.00th=[103285], 99.50th=[162530], 99.90th=[267387], 99.95th=[337642],
     | 99.99th=[566232]
   bw (  KiB/s): min=   15, max= 1936, per=1.01%, avg=632.65, stdev=176.98, samples=198286
   iops        : min=    1, max=  242, avg=79.03, stdev=22.13, samples=198286
  lat (usec)   : 500=0.01%, 750=0.02%, 1000=0.32%
  lat (msec)   : 2=1.21%, 4=2.50%, 10=49.40%, 20=41.95%, 50=2.35%
  lat (msec)   : 100=1.21%, 250=0.91%, 500=0.11%, 750=0.01%, 1000=0.01%
  cpu          : usr=0.07%, sys=0.14%, ctx=7846243, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,7845578,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=61.3MiB/s (64.3MB/s), 61.3MiB/s-61.3MiB/s (64.3MB/s-64.3MB/s), io=59.9GiB (64.3GB), run=1000016-1000016msec
```

## 1 node hdd read
> fio -filename=/mapr/my.cluster.com/hdd/h1 -direct=1 -iodepth 1 -thread -rw=randread -ioengine=psync -bs=8k -size=1T -numjobs=100 -runtime=1000 -group_reporting -name=mytest
```shell=
fio-3.7
Starting 100 threads
mytest: Laying out IO file (1 file / 1048576MiB)
fio: pid=0, err=5/file:filesetup.c:223, func=write, error=Input/output error
Jobs: 31 (f=31): [X(1),_(3),r(1),_(1),r(1),_(4),r(1),E(1),r(1),_(2),r(1),_(8),r(1),_(3),r(1),_(4),r(1),_(1),r(1),_(3),r(1),_(7),r(1),_(1),r(1),_(3),r(1),_(6),r(1),_(1),r(6),_(1),r(1),_(2),r(1),_(5),r(2),_(2),r(1),_(3),r(3),_(3),r(1),_(3),r(1),_(1),r(1)][1.5%][r=114MiB/s,w=0KiB/s][r=14.5k,w=0 IOPS][eta 18h:36m:40s]
mytest: (groupid=0, jobs=100): err= 5 (file:filesetup.c:223, func=write, error=Input/output error): pid=0: Wed Jan 20 22:48:27 2021
   read: IOPS=14.0k, BW=110MiB/s (115MB/s)(107GiB/1000056msec)
    clat (usec): min=101, max=258415, avg=6560.39, stdev=5606.22
     lat (usec): min=102, max=258416, avg=6560.76, stdev=5606.22
    clat percentiles (usec):
     |  1.00th=[  619],  5.00th=[ 2147], 10.00th=[ 3097], 20.00th=[ 3949],
     | 30.00th=[ 4490], 40.00th=[ 4948], 50.00th=[ 5342], 60.00th=[ 5735],
     | 70.00th=[ 6194], 80.00th=[ 6915], 90.00th=[ 9634], 95.00th=[18482],
     | 99.00th=[30016], 99.50th=[36439], 99.90th=[54264], 99.95th=[62653],
     | 99.99th=[84411]
   bw (  KiB/s): min=   16, max= 3072, per=1.09%, avg=1217.79, stdev=154.14, samples=184134
   iops        : min=    2, max=  384, avg=152.18, stdev=19.27, samples=184134
  lat (usec)   : 250=0.05%, 500=0.61%, 750=0.71%, 1000=0.64%
  lat (msec)   : 2=2.56%, 4=16.31%, 10=69.30%, 20=5.80%, 50=3.87%
  lat (msec)   : 100=0.14%, 250=0.01%, 500=0.01%
  cpu          : usr=0.10%, sys=1.26%, ctx=15752693, majf=0, minf=198
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=14028071,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=110MiB/s (115MB/s), 110MiB/s-110MiB/s (115MB/s-115MB/s), io=107GiB (115GB), run=1000056-1000056msec
```


## 3 node ssd write
> fio -filename=/mapr/my.cluster.com/ssd/s1 -direct=1 -iodepth 1 -thread -rw=randread -ioengine=psync -bs=8k -size=1T -numjobs=100 -runtime=1000 -group_reporting -name=mytest
```shell=
mytest: (groupid=0, jobs=100): err= 0: pid=883514: Thu Jan 21 21:52:11 2021
  write: IOPS=7467, BW=58.3MiB/s (61.2MB/s)(56.0GiB/1000013msec)
    clat (usec): min=1879, max=179728, avg=13353.26, stdev=3723.75
     lat (usec): min=1879, max=179728, avg=13353.94, stdev=3723.75
    clat percentiles (usec):
     |  1.00th=[ 7701],  5.00th=[ 8979], 10.00th=[ 9634], 20.00th=[10683],
     | 30.00th=[11469], 40.00th=[12256], 50.00th=[12911], 60.00th=[13698],
     | 70.00th=[14615], 80.00th=[15664], 90.00th=[17171], 95.00th=[18744],
     | 99.00th=[21890], 99.50th=[23725], 99.90th=[53740], 99.95th=[74974],
     | 99.99th=[92799]
   bw (  KiB/s): min=   16, max=  752, per=1.00%, avg=598.64, stdev=56.56, samples=199485
   iops        : min=    2, max=   94, avg=74.78, stdev= 7.08, samples=199485
  lat (msec)   : 2=0.01%, 4=0.01%, 10=12.78%, 20=84.65%, 50=2.46%
  lat (msec)   : 100=0.11%, 250=0.01%
  cpu          : usr=0.07%, sys=0.14%, ctx=7468243, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,7467606,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=58.3MiB/s (61.2MB/s), 58.3MiB/s-58.3MiB/s (61.2MB/s-61.2MB/s), io=56.0GiB (61.2GB), run=1000013-1000013msec
```

## 3 node ssd read
> fio -filename=/mapr/my.cluster.com/ssd/s1 -direct=1 -iodepth 1 -thread -rw=randread -ioengine=psync -bs=8k -size=1T -numjobs=100 -runtime=1000 -group_reporting -name=mytest
```shell=
mytest: Laying out IO file (1 file / 1048576MiB)
fio: pid=0, err=5/file:filesetup.c:223, func=write, error=Input/output error
Jobs: 99 (f=99): [X(1),r(99)][100.0%][r=130MiB/s,w=0KiB/s][r=16.6k,w=0 IOPS][eta 00m:00s]
mytest: (groupid=0, jobs=100): err= 5 (file:filesetup.c:223, func=write, error=Input/output error): pid=0: Thu Jan 21 22:22:59 2021
   read: IOPS=15.4k, BW=120MiB/s (126MB/s)(117GiB/1000007msec)
    clat (usec): min=95, max=35775, avg=5933.30, stdev=2321.72
     lat (usec): min=95, max=35775, avg=5933.62, stdev=2321.72
    clat percentiles (usec):
     |  1.00th=[  693],  5.00th=[ 2114], 10.00th=[ 3556], 20.00th=[ 4490],
     | 30.00th=[ 4948], 40.00th=[ 5276], 50.00th=[ 5669], 60.00th=[ 6194],
     | 70.00th=[ 6652], 80.00th=[ 7308], 90.00th=[ 8455], 95.00th=[10159],
     | 99.00th=[13566], 99.50th=[14615], 99.90th=[16909], 99.95th=[17957],
     | 99.99th=[21103]
   bw (  KiB/s): min=   16, max= 2704, per=1.09%, avg=1346.44, stdev=130.07, samples=182821
   iops        : min=    2, max=  338, avg=168.27, stdev=16.27, samples=182821
  lat (usec)   : 100=0.01%, 250=0.04%, 500=0.47%, 750=0.66%, 1000=0.65%
  lat (msec)   : 2=2.85%, 4=8.56%, 10=81.36%, 20=5.41%, 50=0.02%
  cpu          : usr=0.11%, sys=1.44%, ctx=15802303, majf=0, minf=198
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=15400157,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=120MiB/s (126MB/s), 120MiB/s-120MiB/s (126MB/s-126MB/s), io=117GiB (126GB), run=1000007-1000007msec
```


## 3 node hdd write
> fio -filename=/mapr/my.cluster.com/hdd/h1 -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=8k -size=1T -numjobs=100 -runtime=1000 -group_reporting -name=mytest

```shell=
(groupid=0, jobs=100): err= 0: pid=1248396: Thu Jan 21 22:46:13 2021
  write: IOPS=6176, BW=48.3MiB/s (50.6MB/s)(47.1GiB/1000087msec)
    clat (usec): min=1822, max=2388.4k, avg=16183.54, stdev=32616.01
     lat (usec): min=1823, max=2388.4k, avg=16184.24, stdev=32616.01
    clat percentiles (msec):
     |  1.00th=[    5],  5.00th=[    6], 10.00th=[    7], 20.00th=[    8],
     | 30.00th=[   10], 40.00th=[   11], 50.00th=[   12], 60.00th=[   13],
     | 70.00th=[   14], 80.00th=[   16], 90.00th=[   22], 95.00th=[   40],
     | 99.00th=[  127], 99.50th=[  186], 99.90th=[  409], 99.95th=[  693],
     | 99.99th=[ 1133]
   bw (  KiB/s): min=   15, max= 1184, per=1.02%, avg=502.25, stdev=165.05, samples=196684
   iops        : min=    1, max=  148, avg=62.73, stdev=20.64, samples=196684
  lat (msec)   : 2=0.01%, 4=0.95%, 10=37.63%, 20=50.53%, 50=7.34%
  lat (msec)   : 100=2.17%, 250=1.17%, 500=0.13%, 750=0.04%, 1000=0.02%
  cpu          : usr=0.06%, sys=0.12%, ctx=6177528, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,6177085,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=48.3MiB/s (50.6MB/s), 48.3MiB/s-48.3MiB/s (50.6MB/s-50.6MB/s), io=47.1GiB (50.6GB), run=1000087-1000087msec
```

## 3 node hdd read
> fio -filename=/mapr/my.cluster.com/hdd/h1 -direct=1 -iodepth 1 -thread -rw=randread -ioengine=psync -bs=8k -size=1T -numjobs=100 -runtime=1000 -group_reporting -name=mytest
```shell=
...
fio-3.7
Starting 100 threads
mytest: Laying out IO file (1 file / 1048576MiB)
fio: pid=0, err=5/file:filesetup.c:223, func=write, error=Input/output error
Jobs: 99 (f=99): [X(1),r(99)][100.0%][r=127MiB/s,w=0KiB/s][r=16.3k,w=0 IOPS][eta 00m:00s]
mytest: (groupid=0, jobs=100): err= 5 (file:filesetup.c:223, func=write, error=Input/output error): pid=0: Thu Jan 21 23:15:06 2021
   read: IOPS=15.1k, BW=118MiB/s (124MB/s)(115GiB/1000041msec)
    clat (usec): min=110, max=286385, avg=6092.74, stdev=5479.46
     lat (usec): min=111, max=286385, avg=6093.07, stdev=5479.46
    clat percentiles (usec):
     |  1.00th=[  668],  5.00th=[ 2089], 10.00th=[ 3097], 20.00th=[ 3687],
     | 30.00th=[ 4015], 40.00th=[ 4359], 50.00th=[ 4686], 60.00th=[ 5080],
     | 70.00th=[ 5669], 80.00th=[ 6390], 90.00th=[ 9503], 95.00th=[17433],
     | 99.00th=[29492], 99.50th=[35914], 99.90th=[54264], 99.95th=[63177],
     | 99.99th=[86508]
   bw (  KiB/s): min=   16, max= 2528, per=1.09%, avg=1311.39, stdev=174.91, samples=183930
   iops        : min=    2, max=  316, avg=163.88, stdev=21.87, samples=183930
  lat (usec)   : 250=0.04%, 500=0.51%, 750=0.68%, 1000=0.64%
  lat (msec)   : 2=2.85%, 4=24.51%, 10=61.06%, 20=6.32%, 50=3.24%
  lat (msec)   : 100=0.14%, 250=0.01%, 500=0.01%
  cpu          : usr=0.10%, sys=1.35%, ctx=15451057, majf=0, minf=198
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=15088089,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=118MiB/s (124MB/s), 118MiB/s-118MiB/s (124MB/s-124MB/s), io=115GiB (124GB), run=1000041-1000041msec
```
### 4node hdd write Block=>8k

> fio -filename=/mapr/my.cluster.com/hdd/h1 -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=8k -size=1T -numjobs=100 -runtime=1000 -group_reporting -name=mytest

```shell=
mytest: (groupid=0, jobs=100): err= 0: pid=1560194: Tue Jan 26 21:41:54 2021
  write: IOPS=4234, BW=33.1MiB/s (34.7MB/s)(32.3GiB/1000707msec)
    clat (usec): min=1677, max=4705.1k, avg=23589.82, stdev=75792.02
     lat (usec): min=1677, max=4705.1k, avg=23590.57, stdev=75792.02
    clat percentiles (msec):
     |  1.00th=[    3],  5.00th=[    4], 10.00th=[    5], 20.00th=[    5],
     | 30.00th=[    6], 40.00th=[    7], 50.00th=[   10], 60.00th=[   13],
     | 70.00th=[   16], 80.00th=[   21], 90.00th=[   31], 95.00th=[   62],
     | 99.00th=[  330], 99.50th=[  489], 99.90th=[ 1045], 99.95th=[ 1368],
     | 99.99th=[ 1921]
   bw (  KiB/s): min=   15, max= 1776, per=1.08%, avg=364.31, stdev=194.48, samples=186047
   iops        : min=    1, max=  222, avg=45.49, stdev=24.31, samples=186047
  lat (msec)   : 2=0.02%, 4=9.49%, 10=42.85%, 20=27.26%, 50=14.62%
  lat (msec)   : 100=2.08%, 250=2.17%, 500=1.02%, 750=0.27%, 1000=0.10%
  cpu          : usr=0.05%, sys=0.09%, ctx=4238538, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,4237994,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=33.1MiB/s (34.7MB/s), 33.1MiB/s-33.1MiB/s (34.7MB/s-34.7MB/s), io=32.3GiB (34.7GB), run=1000707-1000707msec
```

### 4node hdd write Block=>256k 

> fio -filename=/mapr/my.cluster.com/hdd/h1 -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=256k -size=1T -numjobs=100 -runtime=1000 -group_reporting -name=mytest

```shell=
mytest: (groupid=0, jobs=100): err= 0: pid=1482875: Tue Jan 26 21:24:06 2021
  write: IOPS=1938, BW=485MiB/s (508MB/s)(473GiB/1000098msec)
    clat (msec): min=2, max=3720, avg=51.57, stdev=64.74
     lat (msec): min=2, max=3720, avg=51.58, stdev=64.74
    clat percentiles (msec):
     |  1.00th=[    6],  5.00th=[    7], 10.00th=[    7], 20.00th=[   10],
     | 30.00th=[   15], 40.00th=[   49], 50.00th=[   64], 60.00th=[   68],
     | 70.00th=[   71], 80.00th=[   75], 90.00th=[   81], 95.00th=[   87],
     | 99.00th=[  120], 99.50th=[  251], 99.90th=[ 1099], 99.95th=[ 1351],
     | 99.99th=[ 1653]
   bw (  KiB/s): min=  509, max=13796, per=1.03%, avg=5106.18, stdev=1483.71, samples=194294
   iops        : min=    1, max=   53, avg=19.86, stdev= 5.80, samples=194294
  lat (msec)   : 4=0.02%, 10=22.47%, 20=12.41%, 50=5.26%, 100=58.14%
  lat (msec)   : 250=1.20%, 500=0.20%, 750=0.10%, 1000=0.07%
  cpu          : usr=0.05%, sys=0.13%, ctx=1938977, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1938405,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=485MiB/s (508MB/s), 485MiB/s-485MiB/s (508MB/s-508MB/s), io=473GiB (508GB), run=1000098-1000098msec
```

### 4node hdd write Block=>8k iodepth=>4

> fio -filename=/mapr/my.cluster.com/hdd/h1 -direct=4 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=8k -size=1T -numjobs=100 -runtime=1000 -group_reporting -name=mytest


```shell=
mytest: (groupid=0, jobs=100): err= 0: pid=1702829: Tue Jan 26 22:14:22 2021
  write: IOPS=1652, BW=413MiB/s (433MB/s)(403GiB/1000063msec)
    clat (msec): min=2, max=8775, avg=60.50, stdev=187.17
     lat (msec): min=2, max=8775, avg=60.51, stdev=187.17
    clat percentiles (msec):
     |  1.00th=[    6],  5.00th=[    7], 10.00th=[    8], 20.00th=[   11],
     | 30.00th=[   16], 40.00th=[   25], 50.00th=[   45], 60.00th=[   65],
     | 70.00th=[   72], 80.00th=[   79], 90.00th=[   86], 95.00th=[   93],
     | 99.00th=[  368], 99.50th=[  995], 99.90th=[ 3239], 99.95th=[ 3809],
     | 99.99th=[ 4597]
   bw (  KiB/s): min=  507, max=20992, per=1.20%, avg=5061.16, stdev=1957.05, samples=167103
   iops        : min=    1, max=   82, avg=19.69, stdev= 7.65, samples=167103
  lat (msec)   : 4=0.09%, 10=19.30%, 20=16.87%, 50=15.95%, 100=44.64%
  lat (msec)   : 250=1.85%, 500=0.48%, 750=0.21%, 1000=0.11%
  cpu          : usr=0.04%, sys=0.11%, ctx=1652917, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1652376,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
  WRITE: bw=413MiB/s (433MB/s), 413MiB/s-413MiB/s (433MB/s-433MB/s), io=403GiB (433GB), run=1000063-1000063msec
```


# dd 測試

## 3 node `dd` hdd write
```shell=
[root@node1 ~]# time dd if=/dev/zero of=/mapr/my.cluster.com/hdd/h1 bs=8k count=1024000
1024000+0 records in
1024000+0 records out
8388608000 bytes (8.4 GB, 7.8 GiB) copied, 8.41648 s, 997 MB/s

real	0m8.605s
user	0m1.102s
sys	0m6.686s
```

## 3 node `dd` hdd read 
```shell=
[root@node1 ~]# time dd if=/mapr/my.cluster.com/hdd/h1 of=/dev/null bs=8k count=1024000
1024000+0 records in
1024000+0 records out
8388608000 bytes (8.4 GB, 7.8 GiB) copied, 8.75784 s, 958 MB/s

real	0m8.763s
user	0m1.594s
sys	0m6.919s
```



## 3 node `dd` ssd write
```shell=
[root@node1 ~]# time dd if=/dev/zero of=/mapr/my.cluster.com/ssd/s1 bs=8k count=1024000
1024000+0 records in
1024000+0 records out
8388608000 bytes (8.4 GB, 7.8 GiB) copied, 8.80219 s, 953 MB/s

real	0m8.828s
user	0m1.020s
sys	0m6.626s
```

## 3 node `dd` ssd read
```shell=
[root@node1 ~]# time dd if=/mapr/my.cluster.com/ssd/s1 of=/dev/null bs=8k count=1024000
1024000+0 records in
1024000+0 records out
8388608000 bytes (8.4 GB, 7.8 GiB) copied, 8.84118 s, 949 MB/s

real	0m8.846s
user	0m1.591s
sys	0m6.974s
```