# simplefs Slab cache remaining in kmem_cache_destroy()
source code repo [simplefs](https://github.com/sysprog21/simplefs)
## Cache leak when remove simplefs module
When we remove the kernel module, the kernel message shows =="Objects remaining in .... on __kmem_cache_shutdown()"==, that means some cache leaks in slab when the module be removed.
```
[97834.762774] BUG simplefs_cache (Tainted: G B W OE ): Objects remaining in simplefs_cache on __kmem_cache_shutdown()
[97834.766579] -----------------------------------------------------------------------------
[97834.771620] Slab 0x00000000cabe2244 objects=12 used=1 fp=0x00000000805ab4f7 flags=0xfffffc0010200(slab|head|node=0|zone=1|lastcpupid=0x1fffff)
[97834.775969] CPU: 0 PID: 28304 Comm: rmmod Tainted: G B W OE 5.15.0-94-generic #104-Ubuntu
[97834.775974] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 0.0.0 02/06/2015
[97834.775976] Call Trace:
[97834.776008] <TASK>
[97834.776016] show_stack+0x52/0x5c
[97834.776023] dump_stack_lvl+0x4a/0x63
[97834.776027] dump_stack+0x10/0x16
[97834.776030] slab_err+0x95/0xcd
[97834.776034] ? cpumask_next+0x23/0x30
[97834.776039] ? irq_work_queue+0x2b/0x50
[97834.776042] __kmem_cache_shutdown.cold+0x3b/0x1af
[97834.776049] kmem_cache_destroy+0x53/0x100
[97834.776053] simplefs_destroy_inode_cache+0x2d/0x48 [simplefs]
[97834.776057] simplefs_exit+0x25/0xe79 [simplefs]
[97834.776060] __do_sys_delete_module.constprop.0+0x184/0x290
[97834.776063] ? do_syscall_64+0x69/0xc0
[97834.776067] ? ksys_read+0x67/0xf0
[97834.776071] __x64_sys_delete_module+0x12/0x20
[97834.776073] do_syscall_64+0x59/0xc0
[97834.776075] ? do_syscall_64+0x69/0xc0
[97834.776077] ? __x64_sys_read+0x19/0x20
[97834.776080] ? do_syscall_64+0x69/0xc0
[97834.776082] ? do_syscall_64+0x69/0xc0
[97834.776084] ? __x64_sys_close+0x11/0x50
[97834.776086] ? do_syscall_64+0x69/0xc0
[97834.776088] ? exc_page_fault+0x89/0x170
[97834.776092] entry_SYSCALL_64_after_hwframe+0x62/0xcc
[97834.776094] RIP: 0033:0x7f708f418aeb
[97834.776108] Code: 73 01 c3 48 8b 0d 45 33 0f 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa b8 b0 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 15 33 0f 00 f7 d8 64 89 01 48
[97834.776110] RSP: 002b:00007ffe7e844138 EFLAGS: 00000206 ORIG_RAX: 00000000000000b0
[97834.776113] RAX: ffffffffffffffda RBX: 000055cd94fdd760 RCX: 00007f708f418aeb
[97834.776115] RDX: 000000000000000a RSI: 0000000000000800 RDI: 000055cd94fdd7c8
[97834.776116] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
[97834.776117] R10: 00007f708f4b0ac0 R11: 0000000000000206 R12: 00007ffe7e844390
[97834.776119] R13: 000055cd94fdd2a0 R14: 00007ffe7e8448f2 R15: 000055cd94fdd760
[97834.776122] </TASK>
[97834.776131] Object 0x00000000690a61cd @offset=4704
[97834.778604] kmem_cache_destroy simplefs_cache: Slab cache still has objects
[97834.781451] CPU: 0 PID: 28304 Comm: rmmod Tainted: G B W OE 5.15.0-94-generic #104-Ubuntu
[97834.781470] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 0.0.0 02/06/2015
[97834.781472] Call Trace:
[97834.781474] <TASK>
[97834.781492] show_stack+0x52/0x5c
[97834.781500] dump_stack_lvl+0x4a/0x63
[97834.781523] dump_stack+0x10/0x16
[97834.781526] kmem_cache_destroy.cold+0x1d/0x22
[97834.781534] simplefs_destroy_inode_cache+0x2d/0x48 [simplefs]
[97834.781540] simplefs_exit+0x25/0xe79 [simplefs]
[97834.781544] __do_sys_delete_module.constprop.0+0x184/0x290
[97834.781548] ? do_syscall_64+0x69/0xc0
[97834.781565] ? ksys_read+0x67/0xf0
[97834.781573] __x64_sys_delete_module+0x12/0x20
[97834.781576] do_syscall_64+0x59/0xc0
[97834.781580] ? do_syscall_64+0x69/0xc0
[97834.781583] ? __x64_sys_read+0x19/0x20
[97834.781587] ? do_syscall_64+0x69/0xc0
[97834.781590] ? do_syscall_64+0x69/0xc0
[97834.781592] ? __x64_sys_close+0x11/0x50
[97834.781595] ? do_syscall_64+0x69/0xc0
[97834.781598] ? exc_page_fault+0x89/0x170
[97834.781602] entry_SYSCALL_64_after_hwframe+0x62/0xcc
[97834.781605] RIP: 0033:0x7f708f418aeb
[97834.781609] Code: 73 01 c3 48 8b 0d 45 33 0f 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa b8 b0 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 15 33 0f 00 f7 d8 64 89 01 48
[97834.781612] RSP: 002b:00007ffe7e844138 EFLAGS: 00000206 ORIG_RAX: 00000000000000b0
[97834.781616] RAX: ffffffffffffffda RBX: 000055cd94fdd760 RCX: 00007f708f418aeb
[97834.781617] RDX: 000000000000000a RSI: 0000000000000800 RDI: 000055cd94fdd7c8
[97834.781619] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
[97834.781620] R10: 00007f708f4b0ac0 R11: 0000000000000206 R12: 00007ffe7e844390
[97834.781622] R13: 000055cd94fdd2a0 R14: 00007ffe7e8448f2 R15: 000055cd94fdd760
[97834.781626] </TASK>
```
you can see that "kmem_cache_destroy simplefs_cache: Slab cache still has objects"
## Finding which cache leak in slab
By [kernel doc](https://www.kernel.org/doc/html/next/filesystems/vfs.html):
* alloc_inode:
this method is called by alloc_inode() to allocate memory for struct inode and initialize it.
* destroy_inode
this method is called by destroy_inode() to release resources allocated for struct inode. It is only required if ->alloc_inode was defined and simply undoes anything done by ->alloc_inode.
So these two hooks are pairs, and by the logs show we still have some cache (means allocating new inode in cache) are not be free in this function.
We can do some tests to check which commond will cause this bug.
by doing
```
touch file1
touch file2
ln file1 hdlink
```
and then umonuting and removing this module, we can easily reproduce this problem.
So there are some problems in ==inode_operations.link==
## root cause
by comparing with [ext4 in kernel](https://elixir.bootlin.com/linux/v6.7.5/source/fs/ext4/namei.c#L3473), the ext4 system will increase ==i_count== after increasing ==__i_nlink==, and the i_count is defeined ["the inode counter indicating how many kernel components use it"](https://linux-kernel-labs.github.io/refs/heads/master/labs/filesystems_part2.html)
### Detail reason
When the module is removed, it calls "kill_block_super", which attempts to free the inode cache. In this function, it traverses all dentries in the superblock list and tries to disconnect the inode and dentry relations. Consequently, when this relation is disconnected, the i_count will be decreased by 1.
Following the above scenario, the i_count of the original file will be decreased twice because it is linked through a hard link. This causes 'i_count' to become -1. Therefore, if 'i_count' is not 0, 'destroy_inode' will not be called in 'evict' function.
### code trace:
[generic_shutdown_super](https://elixir.bootlin.com/linux/v6.7.5/source/fs/super.c#L663) ->
* [shrink_dcache_for_umount](https://elixir.bootlin.com/linux/v6.7.5/source/fs/dcache.c#L1691) ->
* [do_one_tree](https://elixir.bootlin.com/linux/v6.7.5/source/fs/dcache.c#L1680) ->
* [dput](https://elixir.bootlin.com/linux/v6.7.5/source/fs/dcache.c#L895) ->
* [dentry_kill](https://elixir.bootlin.com/linux/v6.7.5/source/fs/dcache.c#L711) ->
* [__dentry_kill](https://elixir.bootlin.com/linux/v6.7.5/source/fs/dcache.c#L578) ->
* [dentry_unlink_inode](https://elixir.bootlin.com/linux/v6.7.5/source/fs/dcache.c#L384) ->
* [iput](https://elixir.bootlin.com/linux/v6.7.5/source/fs/inode.c#L1789)
* [evict_inodes](https://elixir.bootlin.com/linux/v6.7.5/source/fs/inode.c#L713) ->
```c
if (atomic_read(&inode->i_count))
continue;
```
this will make the inode will not be append into dispose_list
* [dispose_list](https://elixir.bootlin.com/linux/v6.7.5/source/fs/inode.c#L691) ->
* [evict](https://elixir.bootlin.com/linux/v6.7.5/source/fs/inode.c#L645)->
* [destroy_inode](https://elixir.bootlin.com/linux/v6.7.5/source/fs/inode.c#L304) ->
```c
if (ops->destroy_inode) {
ops->destroy_inode(inode);
if (!ops->free_inode)
return;
}
```
this hook will not be called. so the cache also do not be free.
## How to fix
```diff
inode_inc_link_count(inode);
+ ihold(inode);
d_instantiate(dentry, inode);
```
## test result
```
[111724.237677] simplefs: module loaded
[111724.261883] loop7: detected capacity change from 0 to 409600
[111724.265119] simplefs: '/dev/loop7' mount success
[111724.349781] simplefs: unmounted disk
[111729.569876] simplefs: module unloaded
```
the bug seems do not happen...
## ref:
https://www.kerneltravel.net/book/book/%E7%AC%AC%E5%85%AB%E7%AB%A0%E8%99%9A%E6%8B%9F%E6%96%87%E4%BB%B6%E7%B3%BB%E7%BB%9F.pdf
https://github.com/histemiss/document/blob/master/vfs/inode.org
https://www.kernel.org/doc/html/next/filesystems/vfs.html
https://linux-kernel-labs.github.io/refs/heads/master/labs/filesystems_part2.html
https://www.kernel.org/doc/html/v5.10/filesystems/caching/fscache.html
Sign in with Wallet
Connect another wallet