# ETCD IO 效能測試

## 安裝 benchmark
* 需先安裝 go 1.21 以上版本

$ SUSEConnect -p sle-module-desktop-applications/15.5/x86_64

$ SUSEConnect -p sle-module-development-tools/15.5/x86_64

$ zypper install go-1.23-150000.3.38.1

$ go version
go version go1.23.1 linux/amd64

* 安裝 benchmark

$ git clone https://github.com/etcd-io/etcd.git; cd etcd

$ go install -v ./tools/benchmark

$ mv $(go list -f "{{.Target}}" ./tools/benchmark) /usr/local/bin/benchmark

$ benchmark --help

## rke2 測試
* put 測試（寫入），這個測試會發送 10000 個 put 操作

$ benchmark --endpoints=127.0.0.1:2379 --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt put --key-size=8 --val-size=256 --total=10000

* range 測試（讀取）

$ benchmark --endpoints=127.0.0.1:2379 --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt range 10000 --consistency=l --total=10000

## rke 測試
* put 測試（寫入），這個測試會發送 10000 個 put 操作

$ benchmark --endpoints=127.0.0.1:2379 --cert /etc/kubernetes/ssl/kube-etcd-192-168-11-102.pem --key /etc/kubernetes/ssl/kube-etcd-192-168-11-102-key.pem --cacert /etc/kubernetes/ssl/kube-ca.pem put --key-size=8 --val-size=256 --total=10000

* range 測試（讀取）

$ benchmark --endpoints=127.0.0.1:2379 --cert /etc/kubernetes/ssl/kube-etcd-192-168-11-102.pem --key /etc/kubernetes/ssl/kube-etcd-192-168-11-102-key.pem --cacert /etc/kubernetes/ssl/kube-ca.pem range 10000 --consistency=l --total=10000

## fio 測試
* 安裝 fio

$ curl -LO https://github.com/rancherlabs/support-tools/raw/master/instant-fio-master/instant-fio-master.sh

$ bash instant-fio-master.sh

# 在 sles 可以直接 install
＄ zypper in fio

* 測試

# rke1 進到以下目錄
$ cd /var/lib/etcd/

# rke2 進到以下目錄
$ cd /var/lib/rancher/rke2/server/db/etcd

$ mkdir test-data

$ fio --rw=write --ioengine=sync --fdatasync=1 --directory=test-data --size=100m --bs=2300 --name=mytest
mytest: (g=0): rw=write, bs=(R) 2300B-2300B, (W) 2300B-2300B, (T) 2300B-2300B, ioengine=sync, iodepth=1
fio-3.23
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=1265KiB/s][w=563 IOPS][eta 00m:00s]
mytest: (groupid=0, jobs=1): err= 0: pid=27216: Tue Jan  7 17:00:52 2025
  write: IOPS=470, BW=1057KiB/s (1082kB/s)(99.0MiB/96919msec); 0 zone resets
    clat (usec): min=8, max=320408, avg=832.45, stdev=3751.66
     lat (usec): min=9, max=320408, avg=833.13, stdev=3751.67
    clat percentiles (usec):
     |  1.00th=[    12],  5.00th=[    15], 10.00th=[    18], 20.00th=[    23],
     | 30.00th=[    28], 40.00th=[    41], 50.00th=[   178], 60.00th=[   229],
     | 70.00th=[   281], 80.00th=[   363], 90.00th=[   734], 95.00th=[  4883],
     | 99.00th=[ 14353], 99.50th=[ 19530], 99.90th=[ 43779], 99.95th=[ 56886],
     | 99.99th=[119014]
   bw (  KiB/s): min=  247, max= 1582, per=100.00%, avg=1057.02, stdev=227.47, samples=193
   iops        : min=  110, max=  704, avg=470.74, stdev=101.27, samples=193
  lat (usec)   : 10=0.32%, 20=15.28%, 50=25.76%, 100=1.35%, 250=21.52%
  lat (usec)   : 500=22.97%, 750=2.95%, 1000=1.59%
  lat (msec)   : 2=1.75%, 4=0.95%, 10=3.62%, 20=1.48%, 50=0.38%
  lat (msec)   : 100=0.05%, 250=0.01%, 500=0.01%
  fsync/fdatasync/sync_file_range:
    sync (usec): min=544, max=136068, avg=1284.51, stdev=2380.34
    sync percentiles (usec):
     |  1.00th=[   652],  5.00th=[   750], 10.00th=[   816], 20.00th=[   914],
     | 30.00th=[  1004], 40.00th=[  1074], 50.00th=[  1156], 60.00th=[  1221],
     | 70.00th=[  1303], 80.00th=[  1369], 90.00th=[  1483], 95.00th=[  1631],
     | 99.00th=[  2540], 99.50th=[  9765], 99.90th=[ 27132], 99.95th=[ 51643],
     | 99.99th=[111674]
  cpu          : usr=0.83%, sys=15.67%, ctx=212612, majf=0, minf=15
  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,45590,0,0 short=45590,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=1057KiB/s (1082kB/s), 1057KiB/s-1057KiB/s (1082kB/s-1082kB/s), io=99.0MiB (105MB), run=96919-96919msec

* fdatasync 的 99.00th 是 2540usec 等於 2.540 毫秒 (ms)，寫入 WAL 檔案時 fdatasync 呼叫的第 99 個百分位必須小於 10 毫秒，就是好的 etcd IO 效能。

環境清除

$ rm -r test-data/

## 優化方式
* 設定 etcd 容器的優先權，使其高於其他 process，但又不能太高導致壓垮伺服器。

$ ionice -c2 -n0 -p `pgrep -x etcd`

## rke1
* rke1 可以直接跑以下 container 來測試

$ docker run --volume /var/lib/etcd/:/var/lib/etcd:Z quay.io/cloud-bulldozer/etcd-perf
......
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}
--------------------------------------------------------------------------------------------------------------------------------------------------------
INFO: 99th percentile of fsync is 3981312 ns
INFO: 99th percentile of the fsync is within the recommended threshold: - 10 ms, the disk can be used to host etcd

## rke2

$ echo 'apiVersion: v1
kind: Pod
metadata:
  name: etcd-perf
spec:
  restartPolicy: Never
  initContainers:
    - name: etcd-perf
      image: quay.io/cloud-bulldozer/etcd-perf
      volumeMounts:
        - name: etcd-data
          mountPath: /var/lib/etcd
  containers:
    - name: wait
      image: busybox
      command: ["sh", "-c", "sleep infinity"]
      lifecycle:
          postStart:
              exec:
                command:
                  - /bin/sh
                  - -c
                  - |
                    rm -rf /var/lib/etcd/etcd_perf.0.0
      volumeMounts:
        - name: etcd-data
          mountPath: /var/lib/etcd
  volumes:
    - name: etcd-data
      hostPath:
        path: /var/lib/rancher/rke2/server/db/etcd
        type: Directory
  tolerations:
    - key: "node-role.kubernetes.io/control-plane"
      operator: "Exists"
      effect: "NoSchedule"
  nodeSelector:
    node-role.kubernetes.io/etcd: "true"' | kubectl apply -f -

* 檢查 log，以 10 ms 為基準，如果超過這個數值就會被判定為 etcd IO 效能太差。

$ kubectl get pod
NAME        READY   STATUS    RESTARTS   AGE
etcd-perf   1/1     Running   0          108s

$ kubectl logs etcd-perf -c etcd-perf
......
INFO: 99th percentile of fsync is 7176192 ns
INFO: 99th percentile of the fsync is within the recommended threshold: - 10 ms, the disk can be used to host etcd

$ kubectl delete pod etcd-perf

## 參考
https://doczhcn.gitbook.io/etcd/index/index-1/performance

https://www.suse.com/support/kb/doc/?id=000020100

https://access.redhat.com/solutions/4885641