---
# System prepended metadata

title: Neuvector 收集 CPU and memory 資源使用與分析
tags: [工具]

---

# Neuvector 收集 CPU and memory 資源使用與分析

## 收集 Neuvector CPU and memory

```
$ kubectl -n cattle-neuvector-system get po
NAME                                                 READY   STATUS      RESTARTS       AGE
neuvector-controller-pod-76c666566b-fjhz8            1/1     Running     0              5d1h
neuvector-enforcer-pod-4dldr                         1/1     Running     0              12d
neuvector-enforcer-pod-9wg2m                         1/1     Running     61 (12d ago)   12d
neuvector-enforcer-pod-fwb29                         1/1     Running     0              12d
neuvector-enforcer-pod-ttf5n                         1/1     Running     0              6d3h
neuvector-enforcer-pod-wtqqs                         1/1     Running     0              12d
neuvector-manager-pod-c49bd4857-dj5b2                1/1     Running     0              12d
neuvector-prometheus-exporter-pod-64bf756454-xnmn8   1/1     Running     7 (21d ago)    28d
neuvector-scanner-pod-5f4588df6c-svzgs               1/1     Running     0              6h42m
neuvector-updater-pod-28895040-5vhnw                 0/1     Completed   0              2d6h
neuvector-updater-pod-28896480-xlws6                 0/1     Completed   0              30h
neuvector-updater-pod-28897920-st89m                 0/1     Completed   0              6h43m

$ kubectl -n cattle-neuvector-system exec -it neuvector-manager-pod-c49bd4857-dj5b2 -- cli
```
* 收集 profile
```
#neuvector-svc-controller.cattle-neuvector-system> login

admin#neuvector-svc-controller.cattle-neuvector-system> show controller
Total controllers: 1
+--------------+---------------------------------------------------------------------------------------------------------------------------------------+-----------+---------+----------------------+-------------+--------+------------------+-----------------+
| id           | name
                               | host_name | version | joined_at            | cluster_ip  | leader | connection_state | disconnected_at |
+--------------+---------------------------------------------------------------------------------------------------------------------------------------+-----------+---------+----------------------+-------------+--------+------------------+-----------------+
| 5b94feb075ec | k8s_neuvector-controller-pod_neuvector-controller-pod-76c666566b-fjhz8_cattle-neuvector-system_2c373517-2d36-458e-afb6-4cbecdec8854_0 | cilium-w1 | v5.4.1  | 2024-12-06T05:19:23Z | 10.42.1.127 | True   | connected        |                 |
+--------------+---------------------------------------------------------------------------------------------------------------------------------------+-----------+---------+----------------------+-------------+--------+------------------+-----------------+

admin#neuvector-svc-controller.cattle-neuvector-system> request controller 5b94feb075ec profile -c all

admin#neuvector-svc-controller.cattle-neuvector-system> show enforcer
Total enforcers: 5
+--------------+-------------------------------------------------------------------------------------------------------------------------+-----------+---------+----------------------+-------------+------------------+-----------------+
| id           | name                                                                                                                    | host_name | version | joined_at            | cluster_ip  | connection_state | disconnected_at |
+--------------+-------------------------------------------------------------------------------------------------------------------------+-----------+---------+----------------------+-------------+------------------+-----------------+
| de046841d67c | k8s_neuvector-enforcer-pod_neuvector-enforcer-pod-4dldr_cattle-neuvector-system_9a243e7b-ef27-439b-992e-af07c03ea5a4_0  | cilium-w2 | v5.4.1  | 2024-11-28T10:02:21Z | 10.42.2.204 | connected        |                 |
| 9a6a4ddca22f | k8s_neuvector-enforcer-pod_neuvector-enforcer-pod-9wg2m_cattle-neuvector-system_3b8faf5e-bb8a-48cb-b567-8a38c4acffcc_61 | cilium-m3 | v5.4.1  | 2024-11-28T15:11:36Z | 10.42.5.220 | connected        |                 |
| e9da4eb63221 | k8s_neuvector-enforcer-pod_neuvector-enforcer-pod-fwb29_cattle-neuvector-system_bb9ccdd0-01c1-4d78-af19-346be43f221a_0  | cilium-w1 | v5.4.1  | 2024-11-28T10:04:19Z | 10.42.1.86  | connected        |                 |
| fe4d07a33e8a | k8s_neuvector-enforcer-pod_neuvector-enforcer-pod-ttf5n_cattle-neuvector-system_4a191bca-dc39-41a8-8c8b-46e4cf02e9d6_0  | cilium-m1 | v5.4.1  | 2024-12-05T03:01:49Z | 10.42.0.223 | connected        |                 |
| 5e6eec9babbb | k8s_neuvector-enforcer-pod_neuvector-enforcer-pod-wtqqs_cattle-neuvector-system_aebd3bdd-ad9c-4f18-ba85-bb87dc4ae834_0  | cilium-m2 | v5.4.1  | 2024-11-28T10:03:42Z | 10.42.4.44  | connected        |                 |
+--------------+-------------------------------------------------------------------------------------------------------------------------+-----------+---------+----------------------+-------------+------------------+-----------------+

admin#neuvector-svc-controller.cattle-neuvector-system> request enforcer de046841d67c profile -c all

admin#neuvector-svc-controller.neuvector> exit
```
* 匯出 profile
```
$ for i in `kubectl get pods -n cattle-neuvector-system | egrep "controller|enforcer" | awk '{print $1}'`;do kubectl -n cattle-neuvector-system cp $i:/var/nv_debug/profile $i.profile; done
```

```
$ ls -l neuvector-controller-pod-76c666566b-fjhz8.profile
total 476
-rw-r--r-- 1 root root  12334 Dec 11 14:50 ctl.cpu.prof
-rw-r--r-- 1 root root   9098 Dec 11 14:50 ctl.goroutine.prof
-rw-r--r-- 1 root root 458711 Dec 11 14:50 ctl.memory.prof

$ ls -l neuvector-enforcer-pod-4dldr.profile
total 192
-rw-r--r-- 1 root root   4822 Dec 11 14:50 enf.cpu.prof
-rw-r--r-- 1 root root   9470 Dec 11 14:50 enf.goroutine.prof
-rw-r--r-- 1 root root 173809 Dec 11 14:50 enf.memory.prof
```

## 分析

* 將收集到的 profile 上傳到 `neu_profile` 目錄裡
* 部屬一個 go container 環境
```
$ ls -l /root/neu_profile/
total 192
-rw-r--r-- 1 root root   4822 Dec 11 16:35 enf.cpu.prof
-rw-r--r-- 1 root root   9470 Dec 11 16:35 enf.goroutine.prof
-rw-r--r-- 1 root root 173809 Dec 11 16:35 enf.memory.prof

$ docker run -itd --name golang-env --net host -v /root/neu_profile:/neu_profile docker.io/taiwanese/golang bash
```
* 進到 container 後啟用網站，需更改自己的 docker host ip 
```
$ docker exec -it golang-env bash

$ cd /neu_profile

$ go tool pprof -http=192.168.11.65:9999 enf.cpu.prof
```
* 透過瀏覽器訪問

![image](https://hackmd.io/_uploads/rJ1NJ08E1e.png)

* 可以查詢是哪個 syscall cpu 資源使用最多

![image](https://hackmd.io/_uploads/Hkm-VhwEyl.png)



* 使用交互模式查詢
* 可以看到 `syscall.Syscall6` 函數使用 cpu 是最多的
* flat：此列顯示特定函數本身所花費的固定 CPU 時間
```
$ go tool pprof enf.cpu.prof
File: agent
Build ID: 51ce6d8ca094085b786718fe71d2a4ed3b56cb08
Type: cpu
Time: Dec 11, 2024 at 1:48am (UTC)
Duration: 30.17s, Total samples = 12.91s (42.79%)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 8280ms, 64.14% of 12910ms total
Dropped 279 nodes (cum <= 64.55ms)
Showing top 10 nodes out of 141
      flat  flat%   sum%        cum   cum%
    2030ms 15.72% 15.72%     2090ms 16.19%  syscall.Syscall6
    1960ms 15.18% 30.91%     2060ms 15.96%  syscall.Syscall
    1520ms 11.77% 42.68%     1520ms 11.77%  runtime.futex
    1340ms 10.38% 53.06%     1890ms 14.64%  runtime.scanobject
     410ms  3.18% 56.24%      410ms  3.18%  runtime.memclrNoHeapPointers
     320ms  2.48% 58.71%      320ms  2.48%  runtime.usleep
     200ms  1.55% 60.26%     1320ms 10.22%  runtime.mallocgc
     180ms  1.39% 61.66%      240ms  1.86%  runtime.findObject
     170ms  1.32% 62.97%      170ms  1.32%  runtime.pageIndexOf (inline)
     150ms  1.16% 64.14%      150ms  1.16%  runtime.nextFreeFast (inline)
```
* 使用 list 指令查看 Syscall6 函數是怎麼被呼叫的
```
(pprof) list Syscall6
Total: 12.91s
ROUTINE ======================== syscall.Syscall6 in /usr/local/go/src/syscall/asm_linux_amd64.s
     2.03s      2.09s (flat, cum) 16.19% of Total
         .          .     36:
         .          .     37:// func rawSyscallNoError(trap, a1, a2, a3 uintptr) (r1, r2 uintptr)
         .          .     38:TEXT ·rawSyscallNoError(SB),NOSPLIT,$0-48
         .          .     39:   MOVQ    a1+8(FP), DI
         .          .     40:   MOVQ    a2+16(FP), SI
         .       20ms     41:   MOVQ    a3+24(FP), DX
         .          .     42:   MOVQ    trap+0(FP), AX  // syscall entry
         .          .     43:   SYSCALL
         .          .     44:   MOVQ    AX, r1+32(FP)
         .          .     45:   MOVQ    DX, r2+40(FP)
         .          .     46:   RET
         .          .     47:
         .          .     48:// func gettimeofday(tv *Timeval) (err uintptr)
      90ms       90ms     49:TEXT ·gettimeofday(SB),NOSPLIT,$0-16
     1.93s      1.93s     50:   MOVQ    tv+0(FP), DI
         .          .     51:   MOVQ    $0, SI
         .          .     52:   MOVQ    runtime·vdsoGettimeofdaySym(SB), AX
         .          .     53:   TESTQ   AX, AX
         .          .     54:   JZ fallback
         .          .     55:   CALL    AX
         .          .     56:ret:
         .          .     57:   CMPQ    AX, $0xfffffffffffff001
         .          .     58:   JLS     ok7
      10ms       10ms     59:   NEGQ    AX
         .          .     60:   MOVQ    AX, err+8(FP)
         .          .     61:   RET
         .       40ms     62:fallback:
         .          .     63:   MOVL    $SYS_gettimeofday, AX
         .          .     64:   SYSCALL
         .          .     65:   JMP ret
         .          .     66:ok7:
         .          .     67:   MOVQ    $0, err+8(FP)
```
## 參考
https://www.suse.com/support/kb/doc/?id=000020921

https://geektutu.com/post/hpg-pprof.html

https://darjun.github.io/2021/06/09/youdontknowgo/pprof/