slab unreasonably huge

Hi community,

we are hit by an interesting scenario, where SUnreclaim as shown belown is unreasonably huge,

the system is ubuntu 18.04 with kernel ` 4.15.0-45-generic #48-Ubuntu SMP Tue Jan 29 16:28:13 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux`

root@mgt11:~# cat /proc/meminfo
MemTotal: 49453148 kB
MemFree: 359464 kB
MemAvailable: 2989316 kB
Buffers: 98780 kB
Cached: 1879328 kB
SwapCached: 0 kB
Active: 15902400 kB
Inactive: 2222224 kB
Active(anon): 15298100 kB
Inactive(anon): 8248 kB
Active(file): 604300 kB
Inactive(file): 2213976 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 11232 kB
Writeback: 0 kB
AnonPages: 16146616 kB
Mapped: 321448 kB
Shmem: 46604 kB
Slab: 30633444 kB
SReclaimable: 408836 kB
SUnreclaim: 30224608 kB
KernelStack: 48496 kB
PageTables: 75236 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 24726572 kB
Committed_AS: 44564412 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
HardwareCorrupted: 0 kB
AnonHugePages: 1406976 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 1580904 kB
DirectMap2M: 48750592 kB

root@mgt11:~# uname -a
Linux mgt11 4.15.0-45-generic #48-Ubuntu SMP Tue Jan 29 16:28:13 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

A closer look at slab usage, yields that lsm_file_cache uses too many rams, why does this happen and how to fix it?

 Active / Total Objects (% used) : 1266685413 / 1267983641 (99.9%)
 Active / Total Slabs (% used) : 7528094 / 7528094 (100.0%)
 Active / Total Caches (% used) : 89 / 129 (69.0%)
 Active / Total Size (% used) : 30298390.34K / 30488645.30K (99.4%)
 Minimum / Average / Maximum Object : 0.01K / 0.02K / 22.88K

  OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
1263739200 1263739200 100% 0.02K 7433760 170 29735040K lsm_file_cache
658048 281925 0% 0.03K 5141 128 20564K kmalloc-32
421888 183013 0% 0.06K 6592 64 26368K kmalloc-64
348870 189998 0% 0.13K 11629 30 46516K kernfs_node_cache
313383 236417 0% 0.19K 7518 42 60144K dentry
303771 303771 100% 0.10K 7789 39 31156K buffer_head
222016 219669 0% 0.06K 3469 64 13876K pid
188048 60692 0% 0.57K 6716 28 107456K radix_tree_node
184118 183014 0% 0.20K 4722 39 37776K vm_area_struct
177660 43452 0% 0.09K 4230 42 16920K kmalloc-96
135746 134443 0% 0.09K 2951 46 11804K anon_vma
134112 131208 0% 0.25K 4193 32 33544K filp
128768 66900 0% 0.02K 503 256 2012K kmalloc-16
112980 112938 0% 0.19K 2690 42 21520K cred_jar
 90112 90112 100% 0.01K 176 512 704K kmalloc-8
 80416 30124 0% 0.12K 2513 32 10052K kmalloc-128
 71130 69827 0% 0.66K 3083 24 49328K proc_inode_cache
 65984 64825 0% 0.12K 2062 32 8248K eventpoll_epi
 65402 50452 0% 0.59K 2517 26 40272K inode_cache
 61992 59986 0% 0.07K 1107 56 4428K eventpoll_pwq
 50064 33766 0% 0.19K 1192 42 9536K kmalloc-192
 46920 46255 0% 0.69K 1020 46 32640K sock_inode_cache
 43869 32027 0% 1.06K 1465 30 46880K ext4_inode_cache