# Neuvector 收集 CPU and memory 資源使用與分析
## 收集 Neuvector CPU and memory
```
$ kubectl -n cattle-neuvector-system get po
NAME READY STATUS RESTARTS AGE
neuvector-controller-pod-76c666566b-fjhz8 1/1 Running 0 5d1h
neuvector-enforcer-pod-4dldr 1/1 Running 0 12d
neuvector-enforcer-pod-9wg2m 1/1 Running 61 (12d ago) 12d
neuvector-enforcer-pod-fwb29 1/1 Running 0 12d
neuvector-enforcer-pod-ttf5n 1/1 Running 0 6d3h
neuvector-enforcer-pod-wtqqs 1/1 Running 0 12d
neuvector-manager-pod-c49bd4857-dj5b2 1/1 Running 0 12d
neuvector-prometheus-exporter-pod-64bf756454-xnmn8 1/1 Running 7 (21d ago) 28d
neuvector-scanner-pod-5f4588df6c-svzgs 1/1 Running 0 6h42m
neuvector-updater-pod-28895040-5vhnw 0/1 Completed 0 2d6h
neuvector-updater-pod-28896480-xlws6 0/1 Completed 0 30h
neuvector-updater-pod-28897920-st89m 0/1 Completed 0 6h43m
$ kubectl -n cattle-neuvector-system exec -it neuvector-manager-pod-c49bd4857-dj5b2 -- cli
```
* 收集 profile
```
#neuvector-svc-controller.cattle-neuvector-system> login
admin#neuvector-svc-controller.cattle-neuvector-system> show controller
Total controllers: 1
+--------------+---------------------------------------------------------------------------------------------------------------------------------------+-----------+---------+----------------------+-------------+--------+------------------+-----------------+
| id | name
| host_name | version | joined_at | cluster_ip | leader | connection_state | disconnected_at |
+--------------+---------------------------------------------------------------------------------------------------------------------------------------+-----------+---------+----------------------+-------------+--------+------------------+-----------------+
| 5b94feb075ec | k8s_neuvector-controller-pod_neuvector-controller-pod-76c666566b-fjhz8_cattle-neuvector-system_2c373517-2d36-458e-afb6-4cbecdec8854_0 | cilium-w1 | v5.4.1 | 2024-12-06T05:19:23Z | 10.42.1.127 | True | connected | |
+--------------+---------------------------------------------------------------------------------------------------------------------------------------+-----------+---------+----------------------+-------------+--------+------------------+-----------------+
admin#neuvector-svc-controller.cattle-neuvector-system> request controller 5b94feb075ec profile -c all
admin#neuvector-svc-controller.cattle-neuvector-system> show enforcer
Total enforcers: 5
+--------------+-------------------------------------------------------------------------------------------------------------------------+-----------+---------+----------------------+-------------+------------------+-----------------+
| id | name | host_name | version | joined_at | cluster_ip | connection_state | disconnected_at |
+--------------+-------------------------------------------------------------------------------------------------------------------------+-----------+---------+----------------------+-------------+------------------+-----------------+
| de046841d67c | k8s_neuvector-enforcer-pod_neuvector-enforcer-pod-4dldr_cattle-neuvector-system_9a243e7b-ef27-439b-992e-af07c03ea5a4_0 | cilium-w2 | v5.4.1 | 2024-11-28T10:02:21Z | 10.42.2.204 | connected | |
| 9a6a4ddca22f | k8s_neuvector-enforcer-pod_neuvector-enforcer-pod-9wg2m_cattle-neuvector-system_3b8faf5e-bb8a-48cb-b567-8a38c4acffcc_61 | cilium-m3 | v5.4.1 | 2024-11-28T15:11:36Z | 10.42.5.220 | connected | |
| e9da4eb63221 | k8s_neuvector-enforcer-pod_neuvector-enforcer-pod-fwb29_cattle-neuvector-system_bb9ccdd0-01c1-4d78-af19-346be43f221a_0 | cilium-w1 | v5.4.1 | 2024-11-28T10:04:19Z | 10.42.1.86 | connected | |
| fe4d07a33e8a | k8s_neuvector-enforcer-pod_neuvector-enforcer-pod-ttf5n_cattle-neuvector-system_4a191bca-dc39-41a8-8c8b-46e4cf02e9d6_0 | cilium-m1 | v5.4.1 | 2024-12-05T03:01:49Z | 10.42.0.223 | connected | |
| 5e6eec9babbb | k8s_neuvector-enforcer-pod_neuvector-enforcer-pod-wtqqs_cattle-neuvector-system_aebd3bdd-ad9c-4f18-ba85-bb87dc4ae834_0 | cilium-m2 | v5.4.1 | 2024-11-28T10:03:42Z | 10.42.4.44 | connected | |
+--------------+-------------------------------------------------------------------------------------------------------------------------+-----------+---------+----------------------+-------------+------------------+-----------------+
admin#neuvector-svc-controller.cattle-neuvector-system> request enforcer de046841d67c profile -c all
admin#neuvector-svc-controller.neuvector> exit
```
* 匯出 profile
```
$ for i in `kubectl get pods -n cattle-neuvector-system | egrep "controller|enforcer" | awk '{print $1}'`;do kubectl -n cattle-neuvector-system cp $i:/var/nv_debug/profile $i.profile; done
```
```
$ ls -l neuvector-controller-pod-76c666566b-fjhz8.profile
total 476
-rw-r--r-- 1 root root 12334 Dec 11 14:50 ctl.cpu.prof
-rw-r--r-- 1 root root 9098 Dec 11 14:50 ctl.goroutine.prof
-rw-r--r-- 1 root root 458711 Dec 11 14:50 ctl.memory.prof
$ ls -l neuvector-enforcer-pod-4dldr.profile
total 192
-rw-r--r-- 1 root root 4822 Dec 11 14:50 enf.cpu.prof
-rw-r--r-- 1 root root 9470 Dec 11 14:50 enf.goroutine.prof
-rw-r--r-- 1 root root 173809 Dec 11 14:50 enf.memory.prof
```
## 分析
* 將收集到的 profile 上傳到 `neu_profile` 目錄裡
* 部屬一個 go container 環境
```
$ ls -l /root/neu_profile/
total 192
-rw-r--r-- 1 root root 4822 Dec 11 16:35 enf.cpu.prof
-rw-r--r-- 1 root root 9470 Dec 11 16:35 enf.goroutine.prof
-rw-r--r-- 1 root root 173809 Dec 11 16:35 enf.memory.prof
$ docker run -itd --name golang-env --net host -v /root/neu_profile:/neu_profile docker.io/taiwanese/golang bash
```
* 進到 container 後啟用網站，需更改自己的 docker host ip
```
$ docker exec -it golang-env bash
$ cd /neu_profile
$ go tool pprof -http=192.168.11.65:9999 enf.cpu.prof
```
* 透過瀏覽器訪問
* 可以查詢是哪個 syscall cpu 資源使用最多
* 使用交互模式查詢
* 可以看到 `syscall.Syscall6` 函數使用 cpu 是最多的
* flat：此列顯示特定函數本身所花費的固定 CPU 時間
```
$ go tool pprof enf.cpu.prof
File: agent
Build ID: 51ce6d8ca094085b786718fe71d2a4ed3b56cb08
Type: cpu
Time: Dec 11, 2024 at 1:48am (UTC)
Duration: 30.17s, Total samples = 12.91s (42.79%)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 8280ms, 64.14% of 12910ms total
Dropped 279 nodes (cum <= 64.55ms)
Showing top 10 nodes out of 141
flat flat% sum% cum cum%
2030ms 15.72% 15.72% 2090ms 16.19% syscall.Syscall6
1960ms 15.18% 30.91% 2060ms 15.96% syscall.Syscall
1520ms 11.77% 42.68% 1520ms 11.77% runtime.futex
1340ms 10.38% 53.06% 1890ms 14.64% runtime.scanobject
410ms 3.18% 56.24% 410ms 3.18% runtime.memclrNoHeapPointers
320ms 2.48% 58.71% 320ms 2.48% runtime.usleep
200ms 1.55% 60.26% 1320ms 10.22% runtime.mallocgc
180ms 1.39% 61.66% 240ms 1.86% runtime.findObject
170ms 1.32% 62.97% 170ms 1.32% runtime.pageIndexOf (inline)
150ms 1.16% 64.14% 150ms 1.16% runtime.nextFreeFast (inline)
```
* 使用 list 指令查看 Syscall6 函數是怎麼被呼叫的
```
(pprof) list Syscall6
Total: 12.91s
ROUTINE ======================== syscall.Syscall6 in /usr/local/go/src/syscall/asm_linux_amd64.s
2.03s 2.09s (flat, cum) 16.19% of Total
. . 36:
. . 37:// func rawSyscallNoError(trap, a1, a2, a3 uintptr) (r1, r2 uintptr)
. . 38:TEXT ·rawSyscallNoError(SB),NOSPLIT,$0-48
. . 39: MOVQ a1+8(FP), DI
. . 40: MOVQ a2+16(FP), SI
. 20ms 41: MOVQ a3+24(FP), DX
. . 42: MOVQ trap+0(FP), AX // syscall entry
. . 43: SYSCALL
. . 44: MOVQ AX, r1+32(FP)
. . 45: MOVQ DX, r2+40(FP)
. . 46: RET
. . 47:
. . 48:// func gettimeofday(tv *Timeval) (err uintptr)
90ms 90ms 49:TEXT ·gettimeofday(SB),NOSPLIT,$0-16
1.93s 1.93s 50: MOVQ tv+0(FP), DI
. . 51: MOVQ $0, SI
. . 52: MOVQ runtime·vdsoGettimeofdaySym(SB), AX
. . 53: TESTQ AX, AX
. . 54: JZ fallback
. . 55: CALL AX
. . 56:ret:
. . 57: CMPQ AX, $0xfffffffffffff001
. . 58: JLS ok7
10ms 10ms 59: NEGQ AX
. . 60: MOVQ AX, err+8(FP)
. . 61: RET
. 40ms 62:fallback:
. . 63: MOVL $SYS_gettimeofday, AX
. . 64: SYSCALL
. . 65: JMP ret
. . 66:ok7:
. . 67: MOVQ $0, err+8(FP)
```
## 參考
https://www.suse.com/support/kb/doc/?id=000020921
https://geektutu.com/post/hpg-pprof.html
https://darjun.github.io/2021/06/09/youdontknowgo/pprof/
Sign in with Wallet
Connect another wallet