# Installing Samsung SSDs (980 Pro PCIe Gen 4 2TB)
###### tags: `LDRD`
## System layout
1. Host server `mu2e-trk-05`, behind `mu2egateway01.fnal.gov` or (`mu2egateway01.fnal.gov`);
2. Gigabyte 3U server PIC gen4 mother boards;
3. Intel(R) Xeon(R) Silver 4310 CPU with 48 threads;
4. total of 62 GB memory;
5. [PCIe Gen 4x16 SSD carrier card](https://www.gigabyte.com/SSD/AORUS-Gen4-AIC-Adaptor#kf)
6. Four SAMSUNG 980 PRO PCIe® 4.0 NVMe SSD 2TB on the carrier card.
```bash=
[root@mu2e-trk-05 dingpf]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
...
nvme0n1 259:2 0 1.8T 0 disk
nvme1n1 259:1 0 1.8T 0 disk
nvme2n1 259:0 0 1.8T 0 disk
nvme3n1 259:4 0 1.8T 0 disk
...
```
## Single SSD read/write test
### EXT4 file system
```bash=
# Create ext4 file system
[root@mu2e-trk-05 dingpf] mkfs.ext4 /dev/nvme0n1
[root@mu2e-trk-05 dingpf]# fio --name=rand-write --ioengine=libaio --iodepth=256 --rw=randwrite --bs=4k --direct=1 --size=100% --numjobs=12 --runtime=60 --filename=/dev/nvme0n1 --group_reporting=1
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
...
fio-3.7
Starting 12 processes
Jobs: 12 (f=12): [w(12)][10.0%][r=0KiB/s,w=3972MiB/s][r=0,w=1017k IOPS][eta 00m:54sJobs: 12 (f=12): [w(12)][11.7%][r=0KiB/s,w=3968MiB/s][r=0,w=1016k IOPS][eta 00m:53sJobs: 12 (f=12): [w(12)][13.3%][r=0KiB/s,w=3979MiB/s][r=0,w=1019k IOPS][eta 00m:52sJobs: 12 (f=12): [w(12)][15.0%][r=0KiB/s,w=4002MiB/s][r=0,w=1025k IOPS][eta 00m:51sJobs: 12 (f=12): [w(12)][16.7%][r=0KiB/s,w=4009MiB/s][r=0,w=1026k IOPS][eta 00m:50sJobs: 12 (f=12): [w(12)][18.3%][r=0KiB/s,w=4007MiB/s][r=0,w=1026k IOPS][eta 00m:49sJobs: 12 (f=12): [w(12)][20.0%][r=0KiB/s,w=4034MiB/s][r=0,w=1033k IOPS][eta 00m:48sJobs: 12 (f=12): [w(12)][21.7%][r=0KiB/s,w=4061MiB/s][r=0,w=1040k IOPS][eta 00m:47sJobs: 12 (f=12): [w(12)][23.3%][r=0KiB/s,w=4086MiB/s][r=0,w=1046k IOPS][eta 00m:46sJobs: 12 (f=12): [w(12)][25.0%][r=0KiB/s,w=4102MiB/s][r=0,w=1050k IOPS][eta 00m:45sJobs: 12 (f=12): [w(12)][26.7%][r=0KiB/s,w=4090MiB/s][r=0,w=1047k IOPS][eta 00m:44sJobs: 12 (f=12): [w(12)][28.3%][r=0KiB/s,w=4105MiB/s][r=0,w=1051k IOPS][eta 00m:43sJobs: 12 (f=12): [w(12)][30.0%][r=0KiB/s,w=4113MiB/s][r=0,w=1053k IOPS][eta 00m:42sJobs: 12 (f=12): [w(12)][31.7%][r=0KiB/s,w=4128MiB/s][r=0,w=1057k IOPS][eta 00m:41sJobs: 12 (f=12): [w(12)][33.3%][r=0KiB/s,w=4124MiB/s][r=0,w=1056k IOPS][eta 00m:40sJobs: 12 (f=12): [w(12)][35.0%][r=0KiB/s,w=4122MiB/s][r=0,w=1055k IOPS][eta 00m:39sJobs: 12 (f=12): [w(12)][36.7%][r=0KiB/s,w=4123MiB/s][r=0,w=1055k IOPS][eta 00m:38sJobs: 12 (f=12): [w(12)][38.3%][r=0KiB/s,w=4136MiB/s][r=0,w=1059k IOPS][eta 00m:37sJobs: 12 (f=12): [w(12)][40.0%][r=0KiB/s,w=4114MiB/s][r=0,w=1053k IOPS][eta 00m:36sJobs: 12 (f=12): [w(12)][41.7%][r=0KiB/s,w=4133MiB/s][r=0,w=1058k IOPS][eta 00m:35sJobs: 12 (f=12): [w(12)][43.3%][r=0KiB/s,w=4133MiB/s][r=0,w=1058k IOPS][eta 00m:34sJobs: 12 (f=12): [w(12)][45.0%][r=0KiB/s,w=4129MiB/s][r=0,w=1057k IOPS][eta 00m:33sJobs: 12 (f=12): [w(12)][46.7%][r=0KiB/s,w=4125MiB/s][r=0,w=1056k IOPS][eta 00m:32sJobs: 12 (f=12): [w(12)][48.3%][r=0KiB/s,w=4116MiB/s][r=0,w=1054k IOPS][eta 00m:31sJobs: 12 (f=12): [w(12)][50.0%][r=0KiB/s,w=4119MiB/s][r=0,w=1055k IOPS][eta 00m:30sJobs: 12 (f=12): [w(12)][51.7%][r=0KiB/s,w=4132MiB/s][r=0,w=1058k IOPS][eta 00m:29sJobs: 12 (f=12): [w(12)][53.3%][r=0KiB/s,w=4119MiB/s][r=0,w=1054k IOPS][eta 00m:28sJobs: 12 (f=12): [w(12)][55.0%][r=0KiB/s,w=4111MiB/s][r=0,w=1052k IOPS][eta 00m:27sJobs: 12 (f=12): [w(12)][56.7%][r=0KiB/s,w=4076MiB/s][r=0,w=1043k IOPS][eta 00m:26sJobs: 12 (f=12): [w(12)][58.3%][r=0KiB/s,w=4101MiB/s][r=0,w=1050k IOPS][eta 00m:25sJobs: 12 (f=12): [w(12)][60.0%][r=0KiB/s,w=4092MiB/s][r=0,w=1048k IOPS][eta 00m:24sJobs: 12 (f=12): [w(12)][61.7%][r=0KiB/s,w=4076MiB/s][r=0,w=1043k IOPS][eta 00m:23sJobs: 12 (f=12): [w(12)][63.3%][r=0KiB/s,w=4069MiB/s][r=0,w=1042k IOPS][eta 00m:22sJobs: 12 (f=12): [w(12)][65.0%][r=0KiB/s,w=4080MiB/s][r=0,w=1044k IOPS][eta 00m:21sJobs: 12 (f=12): [w(12)][66.7%][r=0KiB/s,w=4057MiB/s][r=0,w=1039k IOPS][eta 00m:20sJobs: 12 (f=12): [w(12)][68.3%][r=0KiB/s,w=4067MiB/s][r=0,w=1041k IOPS][eta 00m:19sJobs: 12 (f=12): [w(12)][70.0%][r=0KiB/s,w=4078MiB/s][r=0,w=1044k IOPS][eta 00m:18sJobs: 12 (f=12): [w(12)][71.7%][r=0KiB/s,w=4106MiB/s][r=0,w=1051k IOPS][eta 00m:17sJobs: 12 (f=12): [w(12)][73.3%][r=0KiB/s,w=4089MiB/s][r=0,w=1047k IOPS][eta 00m:16sJobs: 12 (f=12): [w(12)][75.0%][r=0KiB/s,w=4085MiB/s][r=0,w=1046k IOPS][eta 00m:15sJobs: 12 (f=12): [w(12)][76.7%][r=0KiB/s,w=4091MiB/s][r=0,w=1047k IOPS][eta 00m:14sJobs: 12 (f=12): [w(12)][78.3%][r=0KiB/s,w=4105MiB/s][r=0,w=1051k IOPS][eta 00m:13sJobs: 12 (f=12): [w(12)][80.0%][r=0KiB/s,w=4056MiB/s][r=0,w=1038k IOPS][eta 00m:12sJobs: 12 (f=12): [w(12)][81.7%][r=0KiB/s,w=4079MiB/s][r=0,w=1044k IOPS][eta 00m:11sJobs: 12 (f=12): [w(12)][83.3%][r=0KiB/s,w=4085MiB/s][r=0,w=1046k IOPS][eta 00m:10sJobs: 12 (f=12): [w(12)][85.0%][r=0KiB/s,w=4145MiB/s][r=0,w=1061k IOPS][eta 00m:09sJobs: 12 (f=12): [w(12)][86.7%][r=0KiB/s,w=4168MiB/s][r=0,w=1067k IOPS][eta 00m:08sJobs: 12 (f=12): [w(12)][88.3%][r=0KiB/s,w=4161MiB/s][r=0,w=1065k IOPS][eta 00m:07sJobs: 12 (f=12): [w(12)][90.0%][r=0KiB/s,w=4139MiB/s][r=0,w=1060k IOPS][eta 00m:06sJobs: 12 (f=12): [w(12)][91.7%][r=0KiB/s,w=4137MiB/s][r=0,w=1059k IOPS][eta 00m:05sJobs: 12 (f=12): [w(12)][93.3%][r=0KiB/s,w=4137MiB/s][r=0,w=1059k IOPS][eta 00m:04sJobs: 12 (f=12): [w(12)][95.0%][r=0KiB/s,w=4142MiB/s][r=0,w=1060k IOPS][eta 00m:03sJobs: 12 (f=12): [w(12)][96.7%][r=0KiB/s,w=4145MiB/s][r=0,w=1061k IOPS][eta 00m:02sJobs: 12 (f=12): [w(12)][98.3%][r=0KiB/s,w=3593MiB/s][r=0,w=920k IOPS][eta 00m:01s]Jobs: 12 (f=12): [w(12)][100.0%][r=0KiB/s,w=1678MiB/s][r=0,w=430k IOPS][eta 00m:00s]
rand-write: (groupid=0, jobs=12): err= 0: pid=2238: Fri Jun 17 11:23:51 2022
write: IOPS=1031k, BW=4027MiB/s (4222MB/s)(236GiB/60009msec)
slat (nsec): min=1187, max=4528.6k, avg=1764.93, stdev=1555.89
clat (usec): min=11, max=41747, avg=2976.81, stdev=1424.69
lat (usec): min=14, max=41749, avg=2978.65, stdev=1424.75
clat percentiles (usec):
| 1.00th=[ 1844], 5.00th=[ 2114], 10.00th=[ 2147], 20.00th=[ 2180],
| 30.00th=[ 2376], 40.00th=[ 2409], 50.00th=[ 2442], 60.00th=[ 2638],
| 70.00th=[ 2868], 80.00th=[ 2966], 90.00th=[ 4817], 95.00th=[ 6390],
| 99.00th=[ 8717], 99.50th=[ 9241], 99.90th=[13566], 99.95th=[17171],
| 99.99th=[23725]
bw ( KiB/s): min=57648, max=506568, per=8.33%, avg=343603.24, stdev=105240.16, samples=1440
iops : min=14412, max=126642, avg=85900.79, stdev=26310.04, samples=1440
lat (usec) : 20=0.01%, 50=0.41%, 100=0.23%, 250=0.04%, 500=0.06%
lat (usec) : 750=0.05%, 1000=0.05%
lat (msec) : 2=1.28%, 4=81.64%, 10=15.99%, 20=0.23%, 50=0.01%
cpu : usr=12.05%, sys=18.87%, ctx=15409824, majf=0, minf=111212
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=0,61857357,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
WRITE: bw=4027MiB/s (4222MB/s), 4027MiB/s-4027MiB/s (4222MB/s-4222MB/s), io=236GiB (253GB), run=60009-60009msec
Disk stats (read/write):
nvme0n1: ios=43/61793264, merge=0/0, ticks=3/183169609, in_queue=183169612, util=99.95%
```
For reference, the previous generation of SAMSUNG SSD used in the smartSSD yields:
```bash=
Run status group 0 (all jobs):
WRITE: bw=970MiB/s (1017MB/s), 970MiB/s-970MiB/s (1017MB/s-1017MB/s), io=37.8GiB (40.6GB), run=39917-39917msec
Disk stats (read/write):
nvme0n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
```
### XFS file system - This gives similar result as EXT4 test above.
```bash=
[root@mu2e-trk-05 dingpf]# mkfs.xfs /dev/nvme1n1
Discarding blocks...Done.
meta-data=/dev/nvme1n1 isize=512 agcount=4, agsize=122094662 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0, sparse=0
data = bsize=4096 blocks=488378646, imaxpct=5
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=238466, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
[root@mu2e-trk-05 dingpf]# fio --name=rand-write --ioengine=libaio --iodepth=256 --rw=randwrite --bs=4k --direct=1 --size=100% --numjobs=12 --runtime=60 --filename=/dev/nvme1n1 --group_reporting=1
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
...
fio-3.7
Starting 12 processes
Jobs: 12 (f=12): [w(12)][10.0%][r=0KiB/s,w=4143MiB/s][r=0,w=1061k IOPS][eta 00m:54sJobs: 12 (f=12): [w(12)][11.7%][r=0KiB/s,w=4145MiB/s][r=0,w=1061k IOPS][eta 00m:53sJobs: 12 (f=12): [w(12)][13.3%][r=0KiB/s,w=4142MiB/s][r=0,w=1060k IOPS][eta 00m:52sJobs: 12 (f=12): [w(12)][15.0%][r=0KiB/s,w=4138MiB/s][r=0,w=1059k IOPS][eta 00m:51sJobs: 12 (f=12): [w(12)][16.7%][r=0KiB/s,w=4147MiB/s][r=0,w=1062k IOPS][eta 00m:50sJobs: 12 (f=12): [w(12)][18.3%][r=0KiB/s,w=4137MiB/s][r=0,w=1059k IOPS][eta 00m:49sJobs: 12 (f=12): [w(12)][20.0%][r=0KiB/s,w=4128MiB/s][r=0,w=1057k IOPS][eta 00m:48sJobs: 12 (f=12): [w(12)][21.7%][r=0KiB/s,w=4145MiB/s][r=0,w=1061k IOPS][eta 00m:47sJobs: 12 (f=12): [w(12)][23.3%][r=0KiB/s,w=4141MiB/s][r=0,w=1060k IOPS][eta 00m:46sJobs: 12 (f=12): [w(12)][25.0%][r=0KiB/s,w=4146MiB/s][r=0,w=1062k IOPS][eta 00m:45sJobs: 12 (f=12): [w(12)][26.7%][r=0KiB/s,w=4147MiB/s][r=0,w=1062k IOPS][eta 00m:44sJobs: 12 (f=12): [w(12)][28.3%][r=0KiB/s,w=4148MiB/s][r=0,w=1062k IOPS][eta 00m:43sJobs: 12 (f=12): [w(12)][30.0%][r=0KiB/s,w=4133MiB/s][r=0,w=1058k IOPS][eta 00m:42sJobs: 12 (f=12): [w(12)][31.7%][r=0KiB/s,w=4122MiB/s][r=0,w=1055k IOPS][eta 00m:41sJobs: 12 (f=12): [w(12)][33.3%][r=0KiB/s,w=4137MiB/s][r=0,w=1059k IOPS][eta 00m:40sJobs: 12 (f=12): [w(12)][35.0%][r=0KiB/s,w=4146MiB/s][r=0,w=1061k IOPS][eta 00m:39sJobs: 12 (f=12): [w(12)][36.7%][r=0KiB/s,w=4143MiB/s][r=0,w=1061k IOPS][eta 00m:38sJobs: 12 (f=12): [w(12)][38.3%][r=0KiB/s,w=4138MiB/s][r=0,w=1059k IOPS][eta 00m:37sJobs: 12 (f=12): [w(12)][40.0%][r=0KiB/s,w=4132MiB/s][r=0,w=1058k IOPS][eta 00m:36sJobs: 12 (f=12): [w(12)][41.7%][r=0KiB/s,w=4140MiB/s][r=0,w=1060k IOPS][eta 00m:35sJobs: 12 (f=12): [w(12)][43.3%][r=0KiB/s,w=4141MiB/s][r=0,w=1060k IOPS][eta 00m:34sJobs: 12 (f=12): [w(12)][45.0%][r=0KiB/s,w=4122MiB/s][r=0,w=1055k IOPS][eta 00m:33sJobs: 12 (f=12): [w(12)][46.7%][r=0KiB/s,w=4139MiB/s][r=0,w=1060k IOPS][eta 00m:32sJobs: 12 (f=12): [w(12)][48.3%][r=0KiB/s,w=4138MiB/s][r=0,w=1059k IOPS][eta 00m:31sJobs: 12 (f=12): [w(12)][50.0%][r=0KiB/s,w=4140MiB/s][r=0,w=1060k IOPS][eta 00m:30sJobs: 12 (f=12): [w(12)][51.7%][r=0KiB/s,w=4147MiB/s][r=0,w=1062k IOPS][eta 00m:29sJobs: 12 (f=12): [w(12)][53.3%][r=0KiB/s,w=4146MiB/s][r=0,w=1061k IOPS][eta 00m:28sJobs: 12 (f=12): [w(12)][55.0%][r=0KiB/s,w=4146MiB/s][r=0,w=1061k IOPS][eta 00m:27sJobs: 12 (f=12): [w(12)][56.7%][r=0KiB/s,w=4041MiB/s][r=0,w=1034k IOPS][eta 00m:26sJobs: 12 (f=12): [w(12)][58.3%][r=0KiB/s,w=3993MiB/s][r=0,w=1022k IOPS][eta 00m:25sJobs: 12 (f=12): [w(12)][60.0%][r=0KiB/s,w=4055MiB/s][r=0,w=1038k IOPS][eta 00m:24sJobs: 12 (f=12): [w(12)][61.7%][r=0KiB/s,w=4033MiB/s][r=0,w=1032k IOPS][eta 00m:23sJobs: 12 (f=12): [w(12)][63.3%][r=0KiB/s,w=4073MiB/s][r=0,w=1043k IOPS][eta 00m:22sJobs: 12 (f=12): [w(12)][65.0%][r=0KiB/s,w=4082MiB/s][r=0,w=1045k IOPS][eta 00m:21sJobs: 12 (f=12): [w(12)][66.7%][r=0KiB/s,w=4144MiB/s][r=0,w=1061k IOPS][eta 00m:20sJobs: 12 (f=12): [w(12)][68.3%][r=0KiB/s,w=4132MiB/s][r=0,w=1058k IOPS][eta 00m:19sJobs: 12 (f=12): [w(12)][70.0%][r=0KiB/s,w=4147MiB/s][r=0,w=1062k IOPS][eta 00m:18sJobs: 12 (f=12): [w(12)][71.7%][r=0KiB/s,w=4142MiB/s][r=0,w=1060k IOPS][eta 00m:17sJobs: 12 (f=12): [w(12)][73.3%][r=0KiB/s,w=4130MiB/s][r=0,w=1057k IOPS][eta 00m:16sJobs: 12 (f=12): [w(12)][75.0%][r=0KiB/s,w=4136MiB/s][r=0,w=1059k IOPS][eta 00m:15sJobs: 12 (f=12): [w(12)][76.7%][r=0KiB/s,w=4132MiB/s][r=0,w=1058k IOPS][eta 00m:14sJobs: 12 (f=12): [w(12)][78.3%][r=0KiB/s,w=4142MiB/s][r=0,w=1060k IOPS][eta 00m:13sJobs: 12 (f=12): [w(12)][80.0%][r=0KiB/s,w=4130MiB/s][r=0,w=1057k IOPS][eta 00m:12sJobs: 12 (f=12): [w(12)][81.7%][r=0KiB/s,w=4144MiB/s][r=0,w=1061k IOPS][eta 00m:11sJobs: 12 (f=12): [w(12)][83.3%][r=0KiB/s,w=4139MiB/s][r=0,w=1060k IOPS][eta 00m:10sJobs: 12 (f=12): [w(12)][85.0%][r=0KiB/s,w=4140MiB/s][r=0,w=1060k IOPS][eta 00m:09sJobs: 12 (f=12): [w(12)][86.7%][r=0KiB/s,w=4144MiB/s][r=0,w=1061k IOPS][eta 00m:08sJobs: 12 (f=12): [w(12)][88.3%][r=0KiB/s,w=4139MiB/s][r=0,w=1060k IOPS][eta 00m:07sJobs: 12 (f=12): [w(12)][90.0%][r=0KiB/s,w=4136MiB/s][r=0,w=1059k IOPS][eta 00m:06sJobs: 12 (f=12): [w(12)][91.7%][r=0KiB/s,w=4140MiB/s][r=0,w=1060k IOPS][eta 00m:05sJobs: 12 (f=12): [w(12)][93.3%][r=0KiB/s,w=4130MiB/s][r=0,w=1057k IOPS][eta 00m:04sJobs: 12 (f=12): [w(12)][95.0%][r=0KiB/s,w=4140MiB/s][r=0,w=1060k IOPS][eta 00m:03sJobs: 12 (f=12): [w(12)][96.7%][r=0KiB/s,w=4050MiB/s][r=0,w=1037k IOPS][eta 00m:02sJobs: 12 (f=12): [w(12)][98.3%][r=0KiB/s,w=1694MiB/s][r=0,w=434k IOPS][eta 00m:01s]Jobs: 12 (f=12): [w(12)][100.0%][r=0KiB/s,w=1595MiB/s][r=0,w=408k IOPS][eta 00m:00s]
rand-write: (groupid=0, jobs=12): err= 0: pid=6118: Fri Jun 17 11:45:24 2022
write: IOPS=1033k, BW=4033MiB/s (4229MB/s)(236GiB/60012msec)
slat (nsec): min=1250, max=970129, avg=1738.50, stdev=649.29
clat (usec): min=14, max=38291, avg=2971.69, stdev=991.46
lat (usec): min=17, max=38293, avg=2973.51, stdev=991.51
clat percentiles (usec):
| 1.00th=[ 1991], 5.00th=[ 2376], 10.00th=[ 2606], 20.00th=[ 2606],
| 30.00th=[ 2638], 40.00th=[ 2835], 50.00th=[ 2835], 60.00th=[ 2835],
| 70.00th=[ 2835], 80.00th=[ 2868], 90.00th=[ 2868], 95.00th=[ 5211],
| 99.00th=[ 7177], 99.50th=[ 7832], 99.90th=[10945], 99.95th=[13042],
| 99.99th=[19792]
bw ( KiB/s): min=72816, max=528120, per=8.33%, avg=344210.81, stdev=64213.87, samples=1440
iops : min=18204, max=132030, avg=86052.68, stdev=16053.46, samples=1440
lat (usec) : 20=0.01%, 50=0.13%, 100=0.03%, 250=0.02%, 500=0.05%
lat (usec) : 750=0.03%, 1000=0.04%
lat (msec) : 2=0.72%, 4=91.79%, 10=7.06%, 20=0.12%, 50=0.01%
cpu : usr=12.42%, sys=18.55%, ctx=15534332, majf=0, minf=65435
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=0,61966236,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
WRITE: bw=4033MiB/s (4229MB/s), 4033MiB/s-4033MiB/s (4229MB/s-4229MB/s), io=236GiB (254GB), run=60012-60012msec
Disk stats (read/write):
nvme1n1: ios=86/61898489, merge=0/0, ticks=1/183196070, in_queue=183196071, util=99.99%
```
## Software RAID 0 test
### Configuration
#### Wipe out filesystem created during the single ssd read/write test
```bash=
# Wipe out filesystem created during the single ssd read/write test
[root@mu2e-trk-05 dingpf]# wipefs /dev/nvme0n1
offset type
----------------------------------------------------------------
0x438 ext4 [filesystem]
UUID: f03366cf-5398-4b3f-b380-782fea031ef8
[root@mu2e-trk-05 dingpf]# wipefs /dev/nvme1n1
offset type
----------------------------------------------------------------
0x0 xfs [filesystem]
UUID: c74e0844-186a-4f67-a544-905142e34d9c
```
#### Create software RAID 0
```bash=
[root@mu2e-trk-05 dingpf]# mdadm --create --verbose /dev/md0 --level=0 --raid-devices=4 /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1
mdadm: chunk size defaults to 512K
mdadm: /dev/nvme0n1 appears to contain an ext2fs file system
size=1953514584K mtime=Wed Dec 31 18:00:00 1969
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.
```
#### Verify disks in RAID 0
```bash=
[root@mu2e-trk-05 dingpf]# mdadm -E /dev/nvme0n1
/dev/nvme0n1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 5e09bfc4:9af0d8b8:757cb711:603c9616
Name : mu2e-trk-05.fnal.gov:0 (local to host mu2e-trk-05.fnal.gov)
Creation Time : Fri Jun 17 11:52:24 2022
Raid Level : raid0
Raid Devices : 4
Avail Dev Size : 3906764976 sectors (1862.89 GiB 2000.26 GB)
Data Offset : 264192 sectors
Super Offset : 8 sectors
Unused Space : before=264112 sectors, after=0 sectors
State : clean
Device UUID : 9f0da97b:ee8556b2:8fb3a650:c2f5831e
Update Time : Fri Jun 17 11:52:24 2022
Bad Block Log : 512 entries available at offset 8 sectors
Checksum : 7c2c4e1a - correct
Events : 0
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
```
#### Create file system (XFS) and mount the raid array
```bash=
[root@mu2e-trk-05 dingpf]# mkfs.xfs /dev/md0
Discarding blocks...Done.
meta-data=/dev/md0 isize=512 agcount=33, agsize=61043072 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0, sparse=0
data = bsize=4096 blocks=1953382400, imaxpct=5
= sunit=128 swidth=512 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=521728, version=2
= sectsz=512 sunit=8 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
[root@mu2e-trk-05 dingpf]# mkdir /mnt/ssd_raid_0
[root@mu2e-trk-05 dingpf]# mount -t xfs /dev/md0 /mnt/ssd_raid_0
```
#### Save the raid configuration and mouting point
```bash=
[root@mu2e-trk-05 dingpf]# mdadm --detail -scan
ARRAY /dev/md0 metadata=1.2 name=mu2e-trk-05.fnal.gov:0 UUID=5e09bfc4:9af0d8b8:757cb711:603c9616
[root@mu2e-trk-05 dingpf]# mdadm --detail -scan > /etc/mdadm.conf
[root@mu2e-trk-05 dingpf]# blkid /dev/md0
/dev/md0: UUID="2a87bf1a-8c0d-4d88-aa5e-f7429b4a7e6b" TYPE="xfs"
[root@mu2e-trk-05 dingpf]# vim /etc/fstab
# added
# UUID=2a87bf1a-8c0d-4d88-aa5e-f7429b4a7e6b /mnt/ssd_raid_0 xfs defaults 00
```
#### Benchmarking results
1. Using 4k block size, achived 860MB/s;
2. Using 1024K block size, achived 13.7GB/s.
#### Benchmarking details
##### 4k block size `fio --name=seqwrite --rw=write --direct=1 --ioengine=libaio --bs=4k --numjobs=10 --size=40G --runtime=60 -filename=/mnt/ssd_raid_0/bench_mark2.trash --group_reporting=1`
```bash=
[root@mu2e-trk-05 dingpf]# fio --name=seqwrite --rw=write --direct=1 --ioengine=libaio --bs=4k --numjobs=10 --size=40G --runtime=60 -filename=/mnt/ssd_raid_0/bench_ma
rk2.trash --group_reporting=1
seqwrite: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.7
Starting 10 processes
seqwrite: Laying out IO file (1 file / 40960MiB)
Jobs: 10 (f=10): [W(10)][100.0%][r=0KiB/s,w=799MiB/s][r=0,w=205k IOPS][eta 00m:00s]
seqwrite: (groupid=0, jobs=10): err= 0: pid=14652: Fri Jun 17 12:28:07 2022
write: IOPS=210k, BW=820MiB/s (860MB/s)(48.1GiB/60002msec)
slat (usec): min=2, max=369, avg=23.05, stdev=14.56
clat (nsec): min=560, max=7969.9k, avg=23900.13, stdev=18720.36
lat (usec): min=14, max=8039, avg=47.09, stdev=22.52
clat percentiles (nsec):
| 1.00th=[12736], 5.00th=[13248], 10.00th=[13632], 20.00th=[15424],
| 30.00th=[17536], 40.00th=[19584], 50.00th=[21376], 60.00th=[23936],
| 70.00th=[26752], 80.00th=[30592], 90.00th=[37120], 95.00th=[42752],
| 99.00th=[57088], 99.50th=[63232], 99.90th=[75264], 99.95th=[80384],
| 99.99th=[91648]
bw ( KiB/s): min=80944, max=103488, per=10.00%, avg=84004.36, stdev=6546.68, samples=1191
iops : min=20236, max=25872, avg=21001.07, stdev=1636.67, samples=1191
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=42.47%, 50=55.35%
lat (usec) : 100=2.17%, 250=0.01%, 500=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%
cpu : usr=5.12%, sys=22.05%, ctx=27472363, majf=0, minf=1626
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,12599687,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=820MiB/s (860MB/s), 820MiB/s-820MiB/s (860MB/s-860MB/s), io=48.1GiB (51.6GB), run=60002-60002msec
Disk stats (read/write):
md0: ios=0/12577358, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/3149926, aggrmerge=0/0, aggrticks=0/61142, aggrin_queue=61142, aggrutil=29.64%
nvme0n1: ios=0/3150086, merge=0/2, ticks=0/60439, in_queue=60439, util=29.35%
nvme1n1: ios=0/3149454, merge=0/0, ticks=0/60580, in_queue=60580, util=29.38%
nvme2n1: ios=0/3150083, merge=0/0, ticks=0/60441, in_queue=60441, util=29.34%
nvme3n1: ios=0/3150083, merge=0/0, ticks=0/63108, in_queue=63108, util=29.64%
```
##### 1024k block size `fio --name=seqwrite --rw=write --direct=1 --ioengine=libaio --bs=1024k --numjobs=10 --size=40G --runtime=60 -filename=/mnt/ssd_raid_0/bench_mark.trash --group_reporting=1`
```bash=
[root@mu2e-trk-05 dingpf]# fio --name=seqwrite --rw=write --direct=1 --ioengine=lib
aio --bs=1024k --numjobs=10 --size=40G --runtime=60 --filename=/mnt/ssd_raid_0/benc
h_mark.trash --group_reporting=1
seqwrite: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
...
fio-3.7
Starting 10 processes
seqwrite: Laying out IO file (1 file / 40960MiB)
Jobs: 10 (f=10): [W(10)][12.5%][r=0KiB/s,w=12.7GiB/s][r=0,w=13.0k IOPS][eta 00m:28sJobs: 10 (f=10): [W(10)][15.6%][r=0KiB/s,w=12.8GiB/s][r=0,w=13.1k IOPS][eta 00m:27sJobs: 10 (f=10): [W(10)][18.8%][r=0KiB/s,w=12.8GiB/s][r=0,w=13.1k IOPS][eta 00m:26sJobs: 10 (f=10): [W(10)][22.6%][r=0KiB/s,w=12.8GiB/s][r=0,w=13.1k IOPS][eta 00m:24sJobs: 10 (f=10): [W(10)][25.8%][r=0KiB/s,w=12.8GiB/s][r=0,w=13.1k IOPS][eta 00m:23sJobs: 10 (f=10): [W(10)][29.0%][r=0KiB/s,w=12.8GiB/s][r=0,w=13.1k IOPS][eta 00m:22sJobs: 10 (f=10): [W(10)][32.3%][r=0KiB/s,w=12.8GiB/s][r=0,w=13.1k IOPS][eta 00m:21sJobs: 10 (f=10): [W(10)][35.5%][r=0KiB/s,w=12.8GiB/s][r=0,w=13.1k IOPS][eta 00m:20sJobs: 10 (f=10): [W(10)][38.7%][r=0KiB/s,w=12.7GiB/s][r=0,w=12.0k IOPS][eta 00m:19sJobs: 10 (f=10): [W(10)][41.9%][r=0KiB/s,w=12.8GiB/s][r=0,w=13.1k IOPS][eta 00m:18sJobs: 10 (f=10): [W(10)][45.2%][r=0KiB/s,w=12.7GiB/s][r=0,w=13.0k IOPS][eta 00m:17sJobs: 10 (f=10): [W(10)][48.4%][r=0KiB/s,w=12.7GiB/s][r=0,w=12.0k IOPS][eta 00m:16sJobs: 10 (f=10): [W(10)][51.6%][r=0KiB/s,w=12.8GiB/s][r=0,w=13.1k IOPS][eta 00m:15sJobs: 10 (f=10): [W(10)][54.8%][r=0KiB/s,w=12.8GiB/s][r=0,w=13.1k IOPS][eta 00m:14sJobs: 10 (f=10): [W(10)][58.1%][r=0KiB/s,w=12.7GiB/s][r=0,w=13.0k IOPS][eta 00m:13sJobs: 10 (f=10): [W(10)][61.3%][r=0KiB/s,w=12.8GiB/s][r=0,w=13.1k IOPS][eta 00m:12sJobs: 10 (f=10): [W(10)][64.5%][r=0KiB/s,w=12.8GiB/s][r=0,w=13.1k IOPS][eta 00m:11sJobs: 10 (f=10): [W(10)][67.7%][r=0KiB/s,w=12.7GiB/s][r=0,w=13.0k IOPS][eta 00m:10sJobs: 10 (f=10): [W(10)][71.0%][r=0KiB/s,w=12.7GiB/s][r=0,w=13.1k IOPS][eta 00m:09sJobs: 10 (f=10): [W(10)][74.2%][r=0KiB/s,w=12.7GiB/s][r=0,w=13.0k IOPS][eta 00m:08sJobs: 10 (f=10): [W(10)][77.4%][r=0KiB/s,w=12.7GiB/s][r=0,w=12.0k IOPS][eta 00m:07sJobs: 10 (f=10): [W(10)][80.6%][r=0KiB/s,w=12.7GiB/s][r=0,w=13.0k IOPS][eta 00m:06sJobs: 10 (f=10): [W(10)][83.9%][r=0KiB/s,w=12.7GiB/s][r=0,w=12.0k IOPS][eta 00m:05sJobs: 10 (f=10): [W(10)][87.1%][r=0KiB/s,w=12.8GiB/s][r=0,w=13.1k IOPS][eta 00m:04sJobs: 10 (f=10): [W(10)][90.3%][r=0KiB/s,w=12.8GiB/s][r=0,w=13.1k IOPS][eta 00m:03sJobs: 10 (f=10): [W(10)][93.5%][r=0KiB/s,w=12.8GiB/s][r=0,w=13.1k IOPS][eta 00m:02sJobs: 10 (f=10): [W(10)][96.8%][r=0KiB/s,w=12.8GiB/s][r=0,w=13.1k IOPS][eta 00m:01sJobs: 10 (f=10): [W(10)][100.0%][r=0KiB/s,w=12.8GiB/s][r=0,w=13.1k IOPS][eta 00m:00s]
seqwrite: (groupid=0, jobs=10): err= 0: pid=10974: Fri Jun 17 12:08:48 2022
write: IOPS=13.1k, BW=12.7GiB/s (13.7GB/s)(400GiB/31382msec)
slat (usec): min=30, max=1116, avg=62.19, stdev=25.80
clat (usec): min=116, max=10321, avg=703.04, stdev=466.45
lat (usec): min=155, max=10448, avg=765.36, stdev=464.47
clat percentiles (usec):
| 1.00th=[ 204], 5.00th=[ 293], 10.00th=[ 314], 20.00th=[ 343],
| 30.00th=[ 383], 40.00th=[ 445], 50.00th=[ 676], 60.00th=[ 889],
| 70.00th=[ 963], 80.00th=[ 1012], 90.00th=[ 1057], 95.00th=[ 1090],
| 99.00th=[ 1319], 99.50th=[ 4293], 99.90th=[ 4883], 99.95th=[ 5604],
| 99.99th=[ 7767]
bw ( MiB/s): min= 1252, max= 1320, per=10.00%, avg=1305.30, stdev= 9.53, samples=620
iops : min= 1252, max= 1320, avg=1305.29, stdev= 9.53, samples=620
lat (usec) : 250=2.89%, 500=42.44%, 750=5.98%, 1000=26.34%
lat (msec) : 2=21.60%, 4=0.03%, 10=0.73%, 20=0.01%
cpu : usr=3.70%, sys=5.17%, ctx=412535, majf=0, minf=3505
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,409600,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=12.7GiB/s (13.7GB/s), 12.7GiB/s-12.7GiB/s (13.7GB/s-13.7GB/s), io=400GiB (429GB), run=31382-31382msec
Disk stats (read/write):
md0: ios=0/1625799, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/409601, aggrmerge=0/0, aggrticks=0/226363, aggrin_queue=226363, aggrutil=98.13%
nvme0n1: ios=0/409602, merge=0/0, ticks=0/206183, in_queue=206183, util=98.10%
nvme1n1: ios=0/409601, merge=0/0, ticks=0/176974, in_queue=176974, util=98.12%
nvme2n1: ios=0/409601, merge=0/0, ticks=0/229158, in_queue=229158, util=97.84%
nvme3n1: ios=0/409601, merge=0/0, ticks=0/293138, in_queue=293138, util=98.13%
```
## Next step
Next step is to perform similar tests as shown [here](https://hackmd.io/fkXgBQ2xR-CFnob3clI3FA).
Installed the following packages from SuperMicro for IPMI configuration:
```bash=
IPMICFG_1.33.2_build.220126
IPMIView_2.20.0_build.220309_bundleJRE_Linux_x64
SMCIPMITool_2.26.0_build.220209_bundleJRE_Linux_x64
TAS_1
```
Sign in with Wallet
Connect another wallet