2

I have an Ubuntu 20.04.4 server with 32GB RAM. The server is running a bunch of LXD containers and two VMs (libvirt+qemu+kvm).

After startup, with all services running, the RAM utilization is about ~12GB.

After 3-4 weeks the RAM utilization reaches ~90%.
If I stop all containers and VMs the utilization is still ~20GB.
However, I cannot figure out who is claiming this memory.
I have already tried clearing the cache, but that doesn't change much.
I compiled the kernel with support for kmemleak but it did not detect anything useful but shows up in slabtop.

systemd-cgtop:

/                                                                                          593      -    23.7G        -        -
machine.slice                                                                                -      -     1.4G        -        -
system.slice                                                                               116      -   301.1M        -        -
user.slice                                                                                  11      -   141.9M        -        -
user.slice/user-1000.slice                                                                  11      -   121.6M        -        -
system.slice/systemd-journald.service                                                        1      -    83.8M        -        -
user.slice/user-1000.slice/session-297429.scope                                              5      -    81.0M        -        -
system.slice/libvirtd.service                                                               22      -    46.2M        -        -
user.slice/user-1000.slice/[email protected]                                                 6      -    39.8M        -        -
system.slice/snapd.service                                                                  36      -    19.8M        -        -
system.slice/cron.service                                                                    1      -    19.3M        -        -
init.scope                                                                                   1      -    14.0M        -        -
system.slice/systemd-udevd.service                                                           1      -    13.2M        -        -
system.slice/multipathd.service                                                              7      -    10.8M        -        -
system.slice/NetworkManager.service                                                          3      -     5.8M        -        -
system.slice/networkd-dispatcher.service                                                     1      -     5.4M        -        -
system.slice/ssh.service                                                                     1      -     5.0M        -        -
system.slice/ModemManager.service                                                            3      -     4.5M        -        -
system.slice/systemd-networkd.service                                                        1      -     3.5M        -        -
system.slice/accounts-daemon.service                                                         3      -     3.5M        -        -
system.slice/udisks2.service                                                                 5      -     3.4M        -        -
system.slice/polkit.service                                                                  3      -     3.0M        -        -
system.slice/rsyslog.service                                                                 4      -     2.8M        -        -
system.slice/systemd-resolved.service                                                        1      -     2.4M        -        -
system.slice/unattended-upgrades.service                                                     2      -     1.8M        -        -
system.slice/dbus.service                                                                    1      -     1.8M        -        -
system.slice/systemd-logind.service                                                          1      -     1.7M        -        -
system.slice/smartmontools.service                                                           1      -     1.5M        -        -
system.slice/systemd-machined.service                                                        1      -     1.5M        -        -
system.slice/systemd-timesyncd.service                                                       2      -     1.4M        -        -
system.slice/virtlogd.service                                                                1      -     1.3M        -        -
system.slice/rtkit-daemon.service                                                            3      -     1.2M        -        -

/proc/meminfo:

MemTotal:       32718604 kB
MemFree:        11480728 kB
MemAvailable:   11612788 kB
Buffers:              28 kB
Cached:           144512 kB
SwapCached:       855404 kB
Active:           520504 kB
Inactive:         541588 kB
Active(anon):     441708 kB
Inactive(anon):   484240 kB
Active(file):      78796 kB
Inactive(file):    57348 kB
Unevictable:       18664 kB
Mlocked:           18664 kB
SwapTotal:      33043136 kB
SwapFree:       32031680 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:         94680 kB
Mapped:           126592 kB
Shmem:               660 kB
KReclaimable:     432484 kB
Slab:           10784740 kB
SReclaimable:     432484 kB
SUnreclaim:     10352256 kB
KernelStack:       10512 kB
PageTables:         5052 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    49402436 kB
Committed_AS:    1816364 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      152512 kB
VmallocChunk:          0 kB
Percpu:          8868864 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:    19383100 kB
DirectMap2M:    14053376 kB
DirectMap1G:           0 kB

slabtop:

 Active / Total Objects (% used)    : 30513607 / 33423869 (91.3%)
 Active / Total Slabs (% used)      : 1384092 / 1384092 (100.0%)
 Active / Total Caches (% used)     : 123 / 203 (60.6%)
 Active / Total Size (% used)       : 9965969.20K / 10757454.91K (92.6%)
 Minimum / Average / Maximum Object : 0.01K / 0.32K / 16.00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME                   
27156909 26001970  95%    0.30K 1194104       26   9552832K kmemleak_object        
754624 742232  98%    0.06K  11791       64     47164K kmalloc-64             
654675 278378  42%    0.57K  23382       28    374112K radix_tree_node        
593436 348958  58%    0.08K  11636       51     46544K Acpi-State             
559744 418325  74%    0.03K   4373      128     17492K kmalloc-32             
496320 483104  97%    0.12K  15510       32     62040K kernfs_node_cache      
487104 155952  32%    0.06K   7611       64     30444K vmap_area              
394240 165965  42%    0.14K  14080       28     56320K btrfs_extent_map       
355580 342674  96%    0.09K   7730       46     30920K trace_event_file       
339573 338310  99%    4.00K  42465        8   1358880K kmalloc-4k             
306348 154794  50%    0.19K   7410       42     59280K dentry                 
145931 104400  71%    1.13K  11552       28    369664K btrfs_inode            
137728 137174  99%    0.02K    538      256      2152K kmalloc-16             
112672  74034  65%    0.50K   3671       32     58736K kmalloc-512            
102479  62366  60%    0.30K   4093       26     32744K btrfs_delayed_node     
 68880  66890  97%    2.00K   4305       16    137760K kmalloc-2k             
 66656  48345  72%    0.25K   2083       32     16664K kmalloc-256            
 64110  47818  74%    0.59K   2376       27     38016K inode_cache            
 50176  50176 100%    0.01K     98      512       392K kmalloc-8              
 44710  43744  97%    0.02K    263      170      1052K lsm_file_cache         
 43056  11444  26%    0.25K   1418       32     11344K pool_workqueue         
 36480  29052  79%    0.06K    570       64      2280K kmalloc-rcl-64         
 33920  25846  76%    0.06K    530       64      2120K anon_vma_chain         
 24822  14264  57%    0.19K    832       42      6656K kmalloc-192            
 23552  23552 100%    0.03K    184      128       736K fsnotify_mark_connector
 23517  17994  76%    0.20K    603       39      4824K vm_area_struct         
 19572  14909  76%    0.09K    466       42      1864K kmalloc-rcl-96         
 18262  15960  87%    0.09K    397       46      1588K anon_vma               
 14548  12905  88%    1.00K    459       32     14688K kmalloc-1k             
 14162  14162 100%    0.05K    194       73       776K file_lock_ctx          
 13104  12141  92%    0.09K    312       42      1248K kmalloc-96             
 13062  13062 100%    0.19K    311       42      2488K cred_jar               
 13056  10983  84%    0.12K    408       32      1632K kmalloc-128            
 12192   8922  73%    0.66K    508       24      8128K proc_inode_cache       
 11730  11444  97%    0.69K   1444       46     46208K squashfs_inode_cache   
 11067  11067 100%    0.08K    217       51       868K task_delay_info        
 10752  10752 100%    0.03K     84      128       336K kmemleak_scan_area     
 10656   8666  81%    0.25K    333       32      2664K filp                   
 10252  10252 100%    0.18K    235       44      1880K kvm_mmu_page_header    
 10200  10200 100%    0.05K    120       85       480K ftrace_event_field     
 10176  10176 100%    0.12K    318       32      1272K pid                    
  9906   9906 100%    0.10K    254       39      1016K Acpi-ParseExt          
  9600   9213  95%    0.12K    300       32      1200K kmalloc-rcl-128        
  9520   9520 100%    0.07K    170       56       680K Acpi-Operand           
  8502   8063  94%    0.81K    218       39      6976K sock_inode_cache       
  7733   7733 100%    0.70K    169       46      5408K shmem_inode_cache      
  7392   7231  97%    0.19K    176       42      1408K skbuff_ext_cache       
  6552   6552 100%    0.19K    163       42      1304K kmalloc-rcl-192        
  6480   6480 100%    0.11K    180       36       720K khugepaged_mm_slot     
  6144   6144 100%    0.02K     24      256        96K ep_head                
  5439   5439 100%    0.42K    147       37      2352K btrfs_ordered_extent   
  5248   4981  94%    0.25K    164       32      1312K skbuff_head_cache      
  4792   4117  85%    4.00K    606        8     19392K biovec-max             
  4326   4326 100%    0.19K    103       42       824K proc_dir_entry         
  4125   4125 100%    0.24K    125       33      1000K tw_sock_TCPv6          
  3978   3978 100%    0.10K    102       39       408K buffer_head            
  3975   3769  94%    0.31K    159       25      1272K mnt_cache              
  3328   3200  96%    1.00K    104       32      3328K RAW                    
  3136   3136 100%    1.12K    112       28      3584K signal_cache           
  3072   2560  83%    0.03K     24      128        96K dnotify_struct         
  2910   2820  96%    1.06K     97       30      3104K UNIX                   
  2522   2396  95%    1.19K     97       26      3104K RAWv6                  
  2448   2448 100%    0.04K     24      102        96K pde_opener             
  2400   2400 100%    0.50K     75       32      1200K skbuff_fclone_cache    
  2112   2080  98%    1.00K     66       32      2112K biovec-64              
  1695   1587  93%    2.06K    113       15      3616K sighand_cache          
  1518   1518 100%    0.69K     33       46      1056K files_cache            
  1500   1500 100%    0.31K     60       25       480K nf_conntrack           
  1260    894  70%    6.06K    252        5      8064K task_struct            
  1260   1260 100%    1.06K     42       30      1344K mm_struct              
  1222   1158  94%    2.38K     94       13      3008K TCPv6                  
  1150   1150 100%    0.34K     25       46       400K taskstats              
   924    924 100%    0.56K     33       28       528K task_group             
   888    888 100%    0.21K     24       37       192K file_lock_cache        
   864    864 100%    0.11K     24       36        96K btrfs_trans_handle     
   855    855 100%    2.19K     62       14      1984K TCP                    
   851    851 100%    0.42K     23       37       368K uts_namespace          
   816    816 100%    0.12K     24       34        96K seq_file               
   816    816 100%    0.04K      8      102        32K ext4_extent_status     
   792    792 100%    0.24K     24       33       192K tw_sock_TCP            
   782    782 100%    0.94K     23       34       736K mqueue_inode_cache     
   720    720 100%    0.13K     24       30        96K pid_namespace          
   704    704 100%    0.06K     11       64        44K kmem_cache_node        
   648    648 100%    1.16K     24       27       768K perf_event             
   640    640 100%    0.12K     20       32        80K scsi_sense_cache       
   624    624 100%    0.30K     24       26       192K request_sock_TCP       
   624    624 100%    0.15K     24       26        96K fuse_request           
   596    566  94%    8.00K    149        4      4768K kmalloc-8k             
   576    576 100%    1.31K     24       24       768K UDPv6                  
   494    494 100%    0.30K     19       26       152K request_sock_TCPv6     
   480    480 100%    0.53K     16       30       256K user_namespace         
   432    432 100%    1.15K     16       27       512K ext4_inode_cache       
   416    416 100%    0.25K     13       32       104K kmem_cache             
   416    416 100%    0.61K     16       26       256K hugetlbfs_inode_cache  
   390    390 100%    0.81K     10       39       320K fuse_inode             
   306    306 100%    0.04K      3      102        12K bio_crypt_ctx          
   292    292 100%    0.05K      4       73        16K mbcache                
   260    260 100%    1.56K     13       20       416K bdev_cache             
   256    256 100%    0.02K      1      256         4K jbd2_revoke_table_s    
   232    232 100%    4.00K     29        8       928K names_cache            
   192    192 100%    1.98K     12       16       384K request_queue          
   170    170 100%    0.02K      1      170         4K mod_hash_entries       
   168    168 100%    4.12K     24        7       768K net_namespace          
   155    155 100%    0.26K      5       31        40K numa_policy            
   132    132 100%    0.72K      3       44        96K fat_inode_cache        
   128    128 100%    0.25K      4       32        32K dquot                  
   128    128 100%    0.06K      2       64         8K ext4_io_end            
   108    108 100%    2.61K      9       12       288K x86_emulator           
    84     84 100%    0.19K      2       42        16K ext4_groupinfo_4k      
    68     68 100%    0.12K      2       34         8K jbd2_journal_head      
    68     68 100%    0.12K      2       34         8K abd_t                  
    64     64 100%    8.00K     16        4       512K irq_remap_cache        
    64     64 100%    2.00K      4       16       128K biovec-128             
    63     63 100%    4.06K      9        7       288K x86_fpu                
    56     56 100%    0.07K      1       56         4K fsnotify_mark          
    56     56 100%    0.14K      2       28         8K ext4_allocation_context
    42     42 100%    0.75K      1       42        32K dax_cache              
    40     40 100%    0.20K      1       40         8K ip4-frags              
    36     36 100%    7.86K      9        4       288K kvm_vcpu               
    30     30 100%    1.06K      1       30        32K dmaengine-unmap-128    
    24     24 100%    0.66K      1       24        16K ovl_inode              
    15     15 100%    2.06K      1       15        32K dmaengine-unmap-256    
     6      6 100%   16.00K      3        2        96K zio_buf_comb_16384     
     0      0   0%    0.01K      0      512         0K kmalloc-rcl-8          
     0      0   0%    0.02K      0      256         0K kmalloc-rcl-16         
     0      0   0%    0.03K      0      128         0K kmalloc-rcl-32         
     0      0   0%    0.25K      0       32         0K kmalloc-rcl-256        
     0      0   0%    0.50K      0       32         0K kmalloc-rcl-512        
     0      0   0%    1.00K      0       32         0K kmalloc-rcl-1k         
     0      0   0%    2.00K      0       16         0K kmalloc-rcl-2k         
     0      0   0%    4.00K      0        8         0K kmalloc-rcl-4k         
     0      0   0%    8.00K      0        4         0K kmalloc-rcl-8k         
     0      0   0%    0.09K      0       42         0K dma-kmalloc-96         
     0      0   0%    0.19K      0       42         0K dma-kmalloc-192        
     0      0   0%    0.01K      0      512         0K dma-kmalloc-8          
     0      0   0%    0.02K      0      256         0K dma-kmalloc-16         
     0      0   0%    0.03K      0      128         0K dma-kmalloc-32         
     0      0   0%    0.06K      0       64         0K dma-kmalloc-64         
     0      0   0%    0.12K      0       32         0K dma-kmalloc-128        
     0      0   0%    0.25K      0       32         0K dma-kmalloc-256        
     0      0   0%    0.50K      0       32         0K dma-kmalloc-512        
     0      0   0%    1.00K      0       32         0K dma-kmalloc-1k         
     0      0   0%    2.00K      0       16         0K dma-kmalloc-2k         
     0      0   0%    4.00K      0        8         0K dma-kmalloc-4k         
     0      0   0%    8.00K      0        4         0K dma-kmalloc-8k         
     0      0   0%    0.12K      0       34         0K iint_cache             
     0      0   0%    1.00K      0       32         0K PING                   
     0      0   0%    0.75K      0       42         0K xfrm_state             
     0      0   0%    0.37K      0       43         0K request_sock_subflow   
     0      0   0%    1.81K      0       17         0K MPTCP                  
     0      0   0%    0.62K      0       25         0K dio                    
     0      0   0%    0.19K      0       42         0K userfaultfd_ctx_cache  
     0      0   0%    0.03K      0      128         0K ext4_pending_reservation
     0      0   0%    0.08K      0       51         0K ext4_fc_dentry_update  
     0      0   0%    0.04K      0      102         0K fat_cache              
     0      0   0%    0.81K      0       39         0K ecryptfs_auth_tok_list_item
     0      0   0%    0.02K      0      256         0K ecryptfs_file_cache    
     0      0   0%    0.94K      0       34         0K ecryptfs_inode_cache   
     0      0   0%    2.82K      0       11         0K dm_uevent              
     0      0   0%    3.23K      0        9         0K kcopyd_job             
     0      0   0%    1.19K      0       26         0K PINGv6                 
     0      0   0%    0.18K      0       44         0K ip6-frags              
     0      0   0%    2.00K      0       16         0K MPTCPv6                
     0      0   0%    0.13K      0       30         0K fscrypt_info           
     0      0   0%    0.25K      0       32         0K fsverity_info          
     0      0   0%    1.25K      0       25         0K AF_VSOCK               
     0      0   0%    0.19K      0       42         0K kcf_sreq_cache         
     0      0   0%    0.50K      0       32         0K kcf_areq_cache         
     0      0   0%    0.19K      0       42         0K kcf_context_cache      
     0      0   0%    4.00K      0        8         0K zfs_btree_leaf_cache   
     0      0   0%    0.44K      0       36         0K ddt_entry_cache        
     0      0   0%    1.22K      0       26         0K zio_cache              
     0      0   0%    0.05K      0       85         0K zio_link_cache         
     0      0   0%    0.50K      0       32         0K zio_buf_comb_512       
     0      0   0%    1.00K      0       32         0K zio_buf_comb_1024      
     0      0   0%    1.50K      0       21         0K zio_buf_comb_1536      
     0      0   0%    2.00K      0       16         0K zio_buf_comb_2048      
     0      0   0%    2.50K      0       12         0K zio_buf_comb_2560      
     0      0   0%    3.00K      0       10         0K zio_buf_comb_3072      
     0      0   0%    3.50K      0        9         0K zio_buf_comb_3584      
     0      0   0%    4.00K      0        8         0K zio_buf_comb_4096      
     0      0   0%    8.00K      0        4         0K zio_buf_comb_5120      
     0      0   0%    8.00K      0        4         0K zio_buf_comb_6144      
     0      0   0%    8.00K      0        4         0K zio_buf_comb_7168      
     0      0   0%    8.00K      0        4         0K zio_buf_comb_8192      
     0      0   0%   12.00K      0        2         0K zio_buf_comb_10240     
     0      0   0%   12.00K      0        2         0K zio_buf_comb_12288     
     0      0   0%   16.00K      0        2         0K zio_buf_comb_14336     
     0      0   0%   16.00K      0        2         0K lz4_cache              
     0      0   0%    0.24K      0       33         0K sa_cache               
     0      0   0%    0.96K      0       33         0K dnode_t                
     0      0   0%    0.32K      0       24         0K arc_buf_hdr_t_full     
     0      0   0%    0.38K      0       41         0K arc_buf_hdr_t_full_crypt
     0      0   0%    0.09K      0       42         0K arc_buf_hdr_t_l2only   
     0      0   0%    0.08K      0       51         0K arc_buf_t              
     0      0   0%    0.38K      0       42         0K dmu_buf_impl_t         
     0      0   0%    0.37K      0       43         0K zil_lwb_cache          
     0      0   0%    0.15K      0       26         0K zil_zcw_cache          
     0      0   0%    0.13K      0       30         0K sio_cache_0            
     0      0   0%    0.15K      0       26         0K sio_cache_1            
     0      0   0%    0.16K      0       24         0K sio_cache_2            
     0      0   0%    1.06K      0       30         0K zfs_znode_cache        
     0      0   0%    0.09K      0       46         0K zfs_znode_hold_cache   
3
  • What do you see with cat /sys/kernel/debug/kmemleak? And before you enabled kmemleak with your kernel, what showed up as the top slab?
    – aviro
    Sep 19, 2022 at 17:37
  • @aviro please look at my update Sep 21, 2022 at 13:06
  • You should edit your original question with the information in the answer. You should never use answers to provide additional details. In addition, in your original answer you showed the content of /proc/meminfo after you enabled memleak, and there we saw that that Slab was taking most of the memory, and the top hitter in slab was kmemleak_object. I would like to see the content of /proc/meminfo without enabling memleak.
    – aviro
    Sep 21, 2022 at 15:16

1 Answer 1

1

In the end, it was a problem with the nct6775 driver. It was loaded by the /etc/modules file. After removing it the error disappeared.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .