---
# System prepended metadata

title: System debugging skills
tags: [fd leak, debug, memory leak]

---

# System debugging skills
###### tags: `memory leak`, `fd leak`, `debug`
[toc]

## Preface
In this note, I'd to record some skills to debug for some system issues like memory leak and fd leak.

## Skill
1. Memory utilization
```
Tomas # top
top - 08:15:01 up 44 min,  1 user,  load average: 0.00, 0.00, 0.00
Tasks:  63 total,   1 running,  62 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  1741400 total,  1600560 free,    73184 used,    67656 buff/cache
KiB Swap:        0 total,        0 free,        0 used.  1557120 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
    1 root      20   0   56816   6548   5264 S  0.0  0.4   0:00.95 systemd
    2 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kthreadd
    3 root      20   0       0      0      0 S  0.0  0.0   0:00.02 ksoftirqd/0
    5 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kworker/0:0H
    6 root      20   0       0      0      0 S  0.0  0.0   0:00.04 kworker/u2:0
    7 root      20   0       0      0      0 S  0.0  0.0   0:00.03 rcu_sched
    8 root      20   0       0      0      0 S  0.0  0.0   0:00.00 rcu_bh
    9 root      rt   0       0      0      0 S  0.0  0.0   0:00.00 migration/0
   10 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 lru-add-drain
   11 root      rt   0       0      0      0 S  0.0  0.0   0:00.00 watchdog/0
   12 root      20   0       0      0      0 S  0.0  0.0   0:00.00 cpuhp/0
   13 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kdevtmpfs
   14 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 netns
   15 root      20   0       0      0      0 S  0.0  0.0   0:00.00 khungtaskd
   16 root      20   0       0      0      0 S  0.0  0.0   0:00.00 oom_reaper
   17 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 writeback
   18 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kcompactd0
   
Tomas # ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0  10368   676 ?        Ss   Nov02   0:00 init [3]                  
root         2  0.0  0.0      0     0 ?        S<   Nov02   0:01 [migration/0]
root         3  0.0  0.0      0     0 ?        SN   Nov02   0:00 [ksoftirqd/0]
root         4  0.0  0.0      0     0 ?        S<   Nov02   0:01 [migration/1]
root         5  0.0  0.0      0     0 ?        SN   Nov02   0:00 [ksoftirqd/1]
root         6  0.0  0.0      0     0 ?        S<   Nov02  29:57 [events/0]
root         7  0.0  0.0      0     0 ?        S<   Nov02   0:00 [events/1]
root         8  0.0  0.0      0     0 ?        S<   Nov02   0:00 [khelper]
root        49  0.0  0.0      0     0 ?        S<   Nov02   0:00 [kthread]
root        54  0.0  0.0      0     0 ?        S<   Nov02   0:00 [kblockd/0]
root        55  0.0  0.0      0     0 ?        S<   Nov02   0:00 [kblockd/1]
root        56  0.0  0.0      0     0 ?        S<   Nov02   0:00 [kacpid]

```
2. Storage utilization
```
Tomas # df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 4.5M      4.5M         0 100% /
dev                      30.4M         0     30.4M   0% /dev
/dev/mtdblock3          792.0K     56.0K    736.0K   7% /usr/etc
none                      2.0M         0      2.0M   0% /dev/shm
none                      2.0M      8.0K      2.0M   0% /tmp
none                      8.0M     12.0K      8.0M   0% /home
none                      1.0M    100.0K    924.0K  10% /var
none                      1.0M         0      1.0M   0% /mnt
none                      1.0M     88.0K    936.0K   9% /etc
```
3. Network statistics
```
Tomas # netstat -pal
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:smux            0.0.0.0:*               LISTEN      311/daemon1
tcp        0      0 0.0.0.0:daem2           0.0.0.0:*               LISTEN      308/daem2
tcp        0      0 0.0.0.0:2666            0.0.0.0:*               LISTEN      311/daemon1
tcp        0      0 0.0.0.0:2987            0.0.0.0:*               LISTEN      314/daem3
tcp        0      0 0.0.0.0:6060            0.0.0.0:*               LISTEN      312/daemo3
tcp        0      0 0.0.0.0:502             0.0.0.0:*               LISTEN      322/daemon4
tcp        0      0 0.0.0.0:2777            0.0.0.0:*               LISTEN      313/dae5
tcp        0      0 0.0.0.0:5566            0.0.0.0:*               LISTEN      322/daemon4
tcp        0      0 0.0.0.0:5567            0.0.0.0:*               LISTEN      329/daemon___6
tcp        0      0 0.0.0.0:3999            0.0.0.0:*               LISTEN      325/daemo7
tcp        0      0 127.0.0.1:49600         127.0.0.1:smux          ESTABLISHED 312/daemo3
tcp        0      0 127.0.0.1:49598         127.0.0.1:smux          ESTABLISHED 308/daem2
tcp        0      0 127.0.0.1:49599         127.0.0.1:smux          ESTABLISHED 314/daem3
tcp        0      0 127.0.0.1:smux          127.0.0.1:49600         ESTABLISHED 311/daemon1
tcp        0      0 127.0.0.1:smux          127.0.0.1:49598         ESTABLISHED 311/daemon1
tcp        0      0 127.0.0.1:smux          127.0.0.1:49599         ESTABLISHED 311/daemon1
tcp        0      0 :::http                 :::*                    LISTEN      391/daemon7
tcp        0      0 :::ssh                  :::*                    LISTEN      306/inetd
tcp        0      0 :::telnet               :::*                    LISTEN      306/inetd
tcp        0      0 :::https                :::*                    LISTEN      391/daemon7
udp        0      0 0.0.0.0:5010            0.0.0.0:*                           321/daemon___7
udp        0      0 0.0.0.0:snmp            0.0.0.0:*                           311/daemon1
udp        0      0 0.0.0.0:bootps          0.0.0.0:*                           314/daem3
udp        0      0 0.0.0.0:bootpc          0.0.0.0:*                           314/daem3
udp        0      0 0.0.0.0:tftp            0.0.0.0:*                           306/inetd
udp        0      0 :::snmp                 :::*                                311/daemon1
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node PID/Program name    Path
unix  2      [ ACC ]     STREAM     LISTENING        776 325/daemo7          /var/run/daemo7.vty
unix  2      [ ACC ]     STREAM     LISTENING        785 329/daemon___6     /var/run/daemon___6.vty
unix  11     [ ]         DGRAM                       561 307/syslogd         /tmp/log
unix  2      [ ACC ]     STREAM     LISTENING        573 308/daem2           /var/run/daem8.api
unix  2      [ ACC ]     STREAM     LISTENING        682 308/daem2           /var/run/daem2.vty
unix  2      [ ACC ]     STREAM     LISTENING        696 312/daemo3          /var/run/daemo3.vty
unix  2      [ ACC ]     STREAM     LISTENING        704 313/dae5            /var/run/da9.vty
unix  2      [ ACC ]     STREAM     LISTENING        717 314/daem3           /var/run/daem3.vty
unix  2      [ ACC ]     STREAM     LISTENING        738 311/daemon1           /var/run/daemon1.vty
unix  2      [ ACC ]     STREAM     LISTENING        759 322/daemon4         /var/run/daemon4.vty
unix  3      [ ]         STREAM     CONNECTED       4776 308/daem2           /var/run/daem2.vty
unix  3      [ ]         STREAM     CONNECTED       4771 3094/vtysh          
unix  3      [ ]         STREAM     CONNECTED       4773 329/daemon___6     /var/run/daemon___6.vty
unix  3      [ ]         STREAM     CONNECTED       4770 3094/vtysh          
unix  3      [ ]         STREAM     CONNECTED       4777 322/daemon4         /var/run/daemon4.vty
unix  3      [ ]         STREAM     CONNECTED       4769 3094/vtysh          
unix  3      [ ]         STREAM     CONNECTED       4778 312/daemo3          /var/run/daemo3.vty
unix  3      [ ]         STREAM     CONNECTED       4768 3094/vtysh          
unix  3      [ ]         STREAM     CONNECTED       4772 325/daemo7          /var/run/daemo7.vty
unix  3      [ ]         STREAM     CONNECTED       4767 3094/vtysh          
unix  3      [ ]         STREAM     CONNECTED       4766 314/daem3           /var/run/daem3.vty
unix  3      [ ]         STREAM     CONNECTED       4765 3094/vtysh          
unix  3      [ ]         STREAM     CONNECTED       4774 313/dae5            /var/run/da9.vty
unix  3      [ ]         STREAM     CONNECTED       4764 3094/vtysh          
unix  3      [ ]         STREAM     CONNECTED       4775 311/daemon1           /var/run/daemon1.vty
unix  3      [ ]         STREAM     CONNECTED       4763 3094/vtysh          
unix  2      [ ]         DGRAM                      4757 2873/login          
unix  2      [ ]         DGRAM                      2199 306/inetd           
unix  3      [ ]         STREAM     CONNECTED       2120 308/daem2           /var/run/daem2.vty
unix  3      [ ]         STREAM     CONNECTED       2119 391/daemon7         
unix  3      [ ]         STREAM     CONNECTED       2122 329/daemon___6     /var/run/daemon___6.vty
unix  3      [ ]         STREAM     CONNECTED       2118 391/daemon7         
unix  3      [ ]         STREAM     CONNECTED       2125 322/daemon4         /var/run/daemon4.vty
unix  3      [ ]         STREAM     CONNECTED       2117 391/daemon7         
unix  3      [ ]         STREAM     CONNECTED       2127 312/daemo3          /var/run/daemo3.vty
unix  3      [ ]         STREAM     CONNECTED       2116 391/daemon7         
unix  3      [ ]         STREAM     CONNECTED       2121 325/daemo7          /var/run/daemo7.vty
unix  3      [ ]         STREAM     CONNECTED       2115 391/daemon7         
unix  3      [ ]         STREAM     CONNECTED       2124 314/daem3           /var/run/daem3.vty
unix  3      [ ]         STREAM     CONNECTED       2114 391/daemon7         
unix  3      [ ]         STREAM     CONNECTED       2126 313/dae5            /var/run/da9.vty
unix  3      [ ]         STREAM     CONNECTED       2113 391/daemon7         
unix  3      [ ]         STREAM     CONNECTED       2123 311/daemon1           /var/run/daemon1.vty
unix  3      [ ]         STREAM     CONNECTED       2112 391/daemon7         
unix  2      [ ]         DGRAM                       765 325/daemo7          
unix  2      [ ]         DGRAM                       757 322/daemon4         
unix  2      [ ]         DGRAM                       754 321/daemon___7     
unix  2      [ ]         DGRAM                       736 311/daemon1           
unix  3      [ ]         STREAM     CONNECTED        724 308/daem2           /var/run/daem8.api
unix  3      [ ]         STREAM     CONNECTED        723 312/daemo3          
unix  3      [ ]         STREAM     CONNECTED        708 308/daem2           /var/run/daem8.api
unix  3      [ ]         STREAM     CONNECTED        707 313/dae5            
unix  2      [ ]         DGRAM                       701 313/dae5            
unix  2      [ ]         DGRAM                       694 312/daemo3          
unix  2      [ ]         DGRAM                       565 308/daem2  
```
4. Show kernel ring buffer
```
Tomas # dmesg
```
5. Check logs
```
Tomas # ls -alh /var/log/
```
![](https://i.imgur.com/XTvwwDP.png)

6. opened file descriptors
```
Tomas # ls -al /proc/*/fd | wc -l
432

Tomas # ls -al /proc/123/fd
dr-x------    2         0 Jan 10 16:51 .
dr-xr-xr-x    5         0 Jan  1 00:00 ..
lr-x------    1        64 Jan 10 16:51 0 -> /dev/null
lrwx------    1        64 Jan 10 16:51 1 -> /dev/console
lrwx------    1        64 Jan 10 16:51 10 -> socket:[733]
lrwx------    1        64 Jan 10 16:51 11 -> socket:[734]
lrwx------    1        64 Jan 10 16:51 12 -> socket:[735]
lrwx------    1        64 Jan 10 16:51 13 -> socket:[736]
lrwx------    1        64 Jan 10 16:51 14 -> socket:[737]
lrwx------    1        64 Jan 10 16:51 15 -> socket:[738]
lrwx------    1        64 Jan 10 16:51 16 -> socket:[740]
lrwx------    1        64 Jan 10 16:51 17 -> socket:[741]
lrwx------    1        64 Jan 10 16:51 18 -> socket:[751]
lrwx------    1        64 Jan 10 16:51 19 -> socket:[754]
lrwx------    1        64 Jan 10 16:51 2 -> /dev/console
lrwx------    1        64 Jan 10 16:51 20 -> socket:[757]
lrwx------    1        64 Jan 10 16:51 21 -> socket:[413065]
l-wx------    1        64 Jan 10 16:51 3 -> /var/log/snmpd.log
lr-x------    1        64 Jan 10 16:51 4 -> pipe:[720]
l-wx------    1        64 Jan 10 16:51 5 -> pipe:[720]
lr-x------    1        64 Jan 10 16:51 6 -> pipe:[727]
l-wx------    1        64 Jan 10 16:51 7 -> pipe:[727]
lrwx------    1        64 Jan 10 16:51 8 -> socket:[728]
lrwx------    1        64 Jan 10 16:51 9 -> socket:[729]
```
7. process pseudo-file system
```
Tomas # ls -al /proc/324
dr-xr-xr-x    5         0 Jan  1 00:00 .
dr-xr-xr-x   48         0 Jan  1  1970 ..
-r--------    1         0 Jan  1 05:28 auxv
--w-------    1         0 Jan  1 05:28 clear_refs
-r--r--r--    1         0 Jan  1 00:00 cmdline
lrwxrwxrwx    1         0 Jan  1 05:28 cwd -> /
-r--------    1         0 Jan  1 05:28 environ
lrwxrwxrwx    1         0 Jan  1 05:28 exe -> /bin/abcd
dr-x------    2         0 Jan  1 05:11 fd
dr-x------    2         0 Jan  1 05:28 fdinfo
-r--r--r--    1         0 Jan  1 05:28 maps
-rw-------    1         0 Jan  1 05:28 mem
-r--r--r--    1         0 Jan  1 05:28 mounts
-r--------    1         0 Jan  1 05:28 mountstats
-rw-r--r--    1         0 Jan  1 05:28 oom_adj
-r--r--r--    1         0 Jan  1 05:28 oom_score
lrwxrwxrwx    1         0 Jan  1 05:28 root -> /
-r--r--r--    1         0 Jan  1 05:28 smaps
-r--r--r--    1         0 Jan  1 00:00 stat
-r--r--r--    1         0 Jan  1 05:28 statm
-r--r--r--    1         0 Jan  1 05:28 status
dr-xr-xr-x    3         0 Jan  1 05:28 task
-r--r--r--    1         0 Jan  1 05:28 wchan

/home # cat /proc/324/statm
1182 544 335 294 0 307 0
/home # cat /proc/324/stat
stat    statm   status
/home # cat /proc/324/status
Name:   abcd
State:  R (running)
SleepAVG:       60%
Tgid:   324
Pid:    324
PPid:   1
TracerPid:      0
Uid:    0       0       0       0
Gid:    0       0       0       0
FDSize: 32
Groups:
VmPeak:     4728 kB
VmSize:     4728 kB
VmLck:         0 kB
VmHWM:      2176 kB
VmRSS:      2176 kB
VmData:     1144 kB
VmStk:        84 kB
VmExe:      1176 kB
VmLib:      2128 kB
VmPTE:         8 kB
Threads:        1
SigQ:   0/512
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000080000000
SigIgn: 0000000000001006
SigCgt: 0000000380004000
CapInh: 0000000000000000
CapPrm: 00000000fffffeff
CapEff: 00000000fffffeff


/home # cat /proc/324/stat
324 (abcd) R 1 211 211 0 -1 4194560 638 750 9 5 7153175 855765 14 48 18 0 1
 0 4932 4841472 544 4294967295 32768 1235128 3196980864 3196978664 567644 0 0
4102 16384 0 0 0 17 0 0 0 0
```


## Reference
https://phoenixnap.com/kb/how-to-view-read-linux-log-files