# System debugging skills
###### tags: `memory leak`, `fd leak`, `debug`
[toc]
## Preface
In this note, I'd to record some skills to debug for some system issues like memory leak and fd leak.
## Skill
1. Memory utilization
```
Tomas # top
top - 08:15:01 up 44 min, 1 user, load average: 0.00, 0.00, 0.00
Tasks: 63 total, 1 running, 62 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 1741400 total, 1600560 free, 73184 used, 67656 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 1557120 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 56816 6548 5264 S 0.0 0.4 0:00.95 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:00.02 ksoftirqd/0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
6 root 20 0 0 0 0 S 0.0 0.0 0:00.04 kworker/u2:0
7 root 20 0 0 0 0 S 0.0 0.0 0:00.03 rcu_sched
8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
9 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
10 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 lru-add-drain
11 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
12 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/0
13 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdevtmpfs
14 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 netns
15 root 20 0 0 0 0 S 0.0 0.0 0:00.00 khungtaskd
16 root 20 0 0 0 0 S 0.0 0.0 0:00.00 oom_reaper
17 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 writeback
18 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kcompactd0
Tomas # ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 10368 676 ? Ss Nov02 0:00 init [3]
root 2 0.0 0.0 0 0 ? S< Nov02 0:01 [migration/0]
root 3 0.0 0.0 0 0 ? SN Nov02 0:00 [ksoftirqd/0]
root 4 0.0 0.0 0 0 ? S< Nov02 0:01 [migration/1]
root 5 0.0 0.0 0 0 ? SN Nov02 0:00 [ksoftirqd/1]
root 6 0.0 0.0 0 0 ? S< Nov02 29:57 [events/0]
root 7 0.0 0.0 0 0 ? S< Nov02 0:00 [events/1]
root 8 0.0 0.0 0 0 ? S< Nov02 0:00 [khelper]
root 49 0.0 0.0 0 0 ? S< Nov02 0:00 [kthread]
root 54 0.0 0.0 0 0 ? S< Nov02 0:00 [kblockd/0]
root 55 0.0 0.0 0 0 ? S< Nov02 0:00 [kblockd/1]
root 56 0.0 0.0 0 0 ? S< Nov02 0:00 [kacpid]
```
2. Storage utilization
```
Tomas # df -h
Filesystem Size Used Available Use% Mounted on
/dev/root 4.5M 4.5M 0 100% /
dev 30.4M 0 30.4M 0% /dev
/dev/mtdblock3 792.0K 56.0K 736.0K 7% /usr/etc
none 2.0M 0 2.0M 0% /dev/shm
none 2.0M 8.0K 2.0M 0% /tmp
none 8.0M 12.0K 8.0M 0% /home
none 1.0M 100.0K 924.0K 10% /var
none 1.0M 0 1.0M 0% /mnt
none 1.0M 88.0K 936.0K 9% /etc
```
3. Network statistics
```
Tomas # netstat -pal
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:smux 0.0.0.0:* LISTEN 311/daemon1
tcp 0 0 0.0.0.0:daem2 0.0.0.0:* LISTEN 308/daem2
tcp 0 0 0.0.0.0:2666 0.0.0.0:* LISTEN 311/daemon1
tcp 0 0 0.0.0.0:2987 0.0.0.0:* LISTEN 314/daem3
tcp 0 0 0.0.0.0:6060 0.0.0.0:* LISTEN 312/daemo3
tcp 0 0 0.0.0.0:502 0.0.0.0:* LISTEN 322/daemon4
tcp 0 0 0.0.0.0:2777 0.0.0.0:* LISTEN 313/dae5
tcp 0 0 0.0.0.0:5566 0.0.0.0:* LISTEN 322/daemon4
tcp 0 0 0.0.0.0:5567 0.0.0.0:* LISTEN 329/daemon___6
tcp 0 0 0.0.0.0:3999 0.0.0.0:* LISTEN 325/daemo7
tcp 0 0 127.0.0.1:49600 127.0.0.1:smux ESTABLISHED 312/daemo3
tcp 0 0 127.0.0.1:49598 127.0.0.1:smux ESTABLISHED 308/daem2
tcp 0 0 127.0.0.1:49599 127.0.0.1:smux ESTABLISHED 314/daem3
tcp 0 0 127.0.0.1:smux 127.0.0.1:49600 ESTABLISHED 311/daemon1
tcp 0 0 127.0.0.1:smux 127.0.0.1:49598 ESTABLISHED 311/daemon1
tcp 0 0 127.0.0.1:smux 127.0.0.1:49599 ESTABLISHED 311/daemon1
tcp 0 0 :::http :::* LISTEN 391/daemon7
tcp 0 0 :::ssh :::* LISTEN 306/inetd
tcp 0 0 :::telnet :::* LISTEN 306/inetd
tcp 0 0 :::https :::* LISTEN 391/daemon7
udp 0 0 0.0.0.0:5010 0.0.0.0:* 321/daemon___7
udp 0 0 0.0.0.0:snmp 0.0.0.0:* 311/daemon1
udp 0 0 0.0.0.0:bootps 0.0.0.0:* 314/daem3
udp 0 0 0.0.0.0:bootpc 0.0.0.0:* 314/daem3
udp 0 0 0.0.0.0:tftp 0.0.0.0:* 306/inetd
udp 0 0 :::snmp :::* 311/daemon1
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags Type State I-Node PID/Program name Path
unix 2 [ ACC ] STREAM LISTENING 776 325/daemo7 /var/run/daemo7.vty
unix 2 [ ACC ] STREAM LISTENING 785 329/daemon___6 /var/run/daemon___6.vty
unix 11 [ ] DGRAM 561 307/syslogd /tmp/log
unix 2 [ ACC ] STREAM LISTENING 573 308/daem2 /var/run/daem8.api
unix 2 [ ACC ] STREAM LISTENING 682 308/daem2 /var/run/daem2.vty
unix 2 [ ACC ] STREAM LISTENING 696 312/daemo3 /var/run/daemo3.vty
unix 2 [ ACC ] STREAM LISTENING 704 313/dae5 /var/run/da9.vty
unix 2 [ ACC ] STREAM LISTENING 717 314/daem3 /var/run/daem3.vty
unix 2 [ ACC ] STREAM LISTENING 738 311/daemon1 /var/run/daemon1.vty
unix 2 [ ACC ] STREAM LISTENING 759 322/daemon4 /var/run/daemon4.vty
unix 3 [ ] STREAM CONNECTED 4776 308/daem2 /var/run/daem2.vty
unix 3 [ ] STREAM CONNECTED 4771 3094/vtysh
unix 3 [ ] STREAM CONNECTED 4773 329/daemon___6 /var/run/daemon___6.vty
unix 3 [ ] STREAM CONNECTED 4770 3094/vtysh
unix 3 [ ] STREAM CONNECTED 4777 322/daemon4 /var/run/daemon4.vty
unix 3 [ ] STREAM CONNECTED 4769 3094/vtysh
unix 3 [ ] STREAM CONNECTED 4778 312/daemo3 /var/run/daemo3.vty
unix 3 [ ] STREAM CONNECTED 4768 3094/vtysh
unix 3 [ ] STREAM CONNECTED 4772 325/daemo7 /var/run/daemo7.vty
unix 3 [ ] STREAM CONNECTED 4767 3094/vtysh
unix 3 [ ] STREAM CONNECTED 4766 314/daem3 /var/run/daem3.vty
unix 3 [ ] STREAM CONNECTED 4765 3094/vtysh
unix 3 [ ] STREAM CONNECTED 4774 313/dae5 /var/run/da9.vty
unix 3 [ ] STREAM CONNECTED 4764 3094/vtysh
unix 3 [ ] STREAM CONNECTED 4775 311/daemon1 /var/run/daemon1.vty
unix 3 [ ] STREAM CONNECTED 4763 3094/vtysh
unix 2 [ ] DGRAM 4757 2873/login
unix 2 [ ] DGRAM 2199 306/inetd
unix 3 [ ] STREAM CONNECTED 2120 308/daem2 /var/run/daem2.vty
unix 3 [ ] STREAM CONNECTED 2119 391/daemon7
unix 3 [ ] STREAM CONNECTED 2122 329/daemon___6 /var/run/daemon___6.vty
unix 3 [ ] STREAM CONNECTED 2118 391/daemon7
unix 3 [ ] STREAM CONNECTED 2125 322/daemon4 /var/run/daemon4.vty
unix 3 [ ] STREAM CONNECTED 2117 391/daemon7
unix 3 [ ] STREAM CONNECTED 2127 312/daemo3 /var/run/daemo3.vty
unix 3 [ ] STREAM CONNECTED 2116 391/daemon7
unix 3 [ ] STREAM CONNECTED 2121 325/daemo7 /var/run/daemo7.vty
unix 3 [ ] STREAM CONNECTED 2115 391/daemon7
unix 3 [ ] STREAM CONNECTED 2124 314/daem3 /var/run/daem3.vty
unix 3 [ ] STREAM CONNECTED 2114 391/daemon7
unix 3 [ ] STREAM CONNECTED 2126 313/dae5 /var/run/da9.vty
unix 3 [ ] STREAM CONNECTED 2113 391/daemon7
unix 3 [ ] STREAM CONNECTED 2123 311/daemon1 /var/run/daemon1.vty
unix 3 [ ] STREAM CONNECTED 2112 391/daemon7
unix 2 [ ] DGRAM 765 325/daemo7
unix 2 [ ] DGRAM 757 322/daemon4
unix 2 [ ] DGRAM 754 321/daemon___7
unix 2 [ ] DGRAM 736 311/daemon1
unix 3 [ ] STREAM CONNECTED 724 308/daem2 /var/run/daem8.api
unix 3 [ ] STREAM CONNECTED 723 312/daemo3
unix 3 [ ] STREAM CONNECTED 708 308/daem2 /var/run/daem8.api
unix 3 [ ] STREAM CONNECTED 707 313/dae5
unix 2 [ ] DGRAM 701 313/dae5
unix 2 [ ] DGRAM 694 312/daemo3
unix 2 [ ] DGRAM 565 308/daem2
```
4. Show kernel ring buffer
```
Tomas # dmesg
```
5. Check logs
```
Tomas # ls -alh /var/log/
```
6. opened file descriptors
```
Tomas # ls -al /proc/*/fd | wc -l
432
Tomas # ls -al /proc/123/fd
dr-x------ 2 0 Jan 10 16:51 .
dr-xr-xr-x 5 0 Jan 1 00:00 ..
lr-x------ 1 64 Jan 10 16:51 0 -> /dev/null
lrwx------ 1 64 Jan 10 16:51 1 -> /dev/console
lrwx------ 1 64 Jan 10 16:51 10 -> socket:[733]
lrwx------ 1 64 Jan 10 16:51 11 -> socket:[734]
lrwx------ 1 64 Jan 10 16:51 12 -> socket:[735]
lrwx------ 1 64 Jan 10 16:51 13 -> socket:[736]
lrwx------ 1 64 Jan 10 16:51 14 -> socket:[737]
lrwx------ 1 64 Jan 10 16:51 15 -> socket:[738]
lrwx------ 1 64 Jan 10 16:51 16 -> socket:[740]
lrwx------ 1 64 Jan 10 16:51 17 -> socket:[741]
lrwx------ 1 64 Jan 10 16:51 18 -> socket:[751]
lrwx------ 1 64 Jan 10 16:51 19 -> socket:[754]
lrwx------ 1 64 Jan 10 16:51 2 -> /dev/console
lrwx------ 1 64 Jan 10 16:51 20 -> socket:[757]
lrwx------ 1 64 Jan 10 16:51 21 -> socket:[413065]
l-wx------ 1 64 Jan 10 16:51 3 -> /var/log/snmpd.log
lr-x------ 1 64 Jan 10 16:51 4 -> pipe:[720]
l-wx------ 1 64 Jan 10 16:51 5 -> pipe:[720]
lr-x------ 1 64 Jan 10 16:51 6 -> pipe:[727]
l-wx------ 1 64 Jan 10 16:51 7 -> pipe:[727]
lrwx------ 1 64 Jan 10 16:51 8 -> socket:[728]
lrwx------ 1 64 Jan 10 16:51 9 -> socket:[729]
```
7. process pseudo-file system
```
Tomas # ls -al /proc/324
dr-xr-xr-x 5 0 Jan 1 00:00 .
dr-xr-xr-x 48 0 Jan 1 1970 ..
-r-------- 1 0 Jan 1 05:28 auxv
--w------- 1 0 Jan 1 05:28 clear_refs
-r--r--r-- 1 0 Jan 1 00:00 cmdline
lrwxrwxrwx 1 0 Jan 1 05:28 cwd -> /
-r-------- 1 0 Jan 1 05:28 environ
lrwxrwxrwx 1 0 Jan 1 05:28 exe -> /bin/abcd
dr-x------ 2 0 Jan 1 05:11 fd
dr-x------ 2 0 Jan 1 05:28 fdinfo
-r--r--r-- 1 0 Jan 1 05:28 maps
-rw------- 1 0 Jan 1 05:28 mem
-r--r--r-- 1 0 Jan 1 05:28 mounts
-r-------- 1 0 Jan 1 05:28 mountstats
-rw-r--r-- 1 0 Jan 1 05:28 oom_adj
-r--r--r-- 1 0 Jan 1 05:28 oom_score
lrwxrwxrwx 1 0 Jan 1 05:28 root -> /
-r--r--r-- 1 0 Jan 1 05:28 smaps
-r--r--r-- 1 0 Jan 1 00:00 stat
-r--r--r-- 1 0 Jan 1 05:28 statm
-r--r--r-- 1 0 Jan 1 05:28 status
dr-xr-xr-x 3 0 Jan 1 05:28 task
-r--r--r-- 1 0 Jan 1 05:28 wchan
/home # cat /proc/324/statm
1182 544 335 294 0 307 0
/home # cat /proc/324/stat
stat statm status
/home # cat /proc/324/status
Name: abcd
State: R (running)
SleepAVG: 60%
Tgid: 324
Pid: 324
PPid: 1
TracerPid: 0
Uid: 0 0 0 0
Gid: 0 0 0 0
FDSize: 32
Groups:
VmPeak: 4728 kB
VmSize: 4728 kB
VmLck: 0 kB
VmHWM: 2176 kB
VmRSS: 2176 kB
VmData: 1144 kB
VmStk: 84 kB
VmExe: 1176 kB
VmLib: 2128 kB
VmPTE: 8 kB
Threads: 1
SigQ: 0/512
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000080000000
SigIgn: 0000000000001006
SigCgt: 0000000380004000
CapInh: 0000000000000000
CapPrm: 00000000fffffeff
CapEff: 00000000fffffeff
/home # cat /proc/324/stat
324 (abcd) R 1 211 211 0 -1 4194560 638 750 9 5 7153175 855765 14 48 18 0 1
0 4932 4841472 544 4294967295 32768 1235128 3196980864 3196978664 567644 0 0
4102 16384 0 0 0 17 0 0 0 0
```
## Reference
https://phoenixnap.com/kb/how-to-view-read-linux-log-files
Sign in with Wallet
Connect another wallet