2

What is using all my cache and why can't I clear it?

I'm not trying to optimise performance or tune the system, the best description is that this is an exercise in understanding and learning and I'm asking that we can just go with it.

Running kernel 4.16.13-200.fc27.x86_64

If I run the following command (which produces the following output)

free;sync;free; echo 3 > /proc/sys/vm/drop_caches;free

                  total        used        free      shared  buff/cache   available
    Mem:       16276472      259676     7034800       59720     8981996     8502360
    Swap:             0           0           0
                  total        used        free      shared  buff/cache   available
    Mem:       16276472      259792     7034652       59720     8982028     8502228
    Swap:             0           0           0
                  total        used        free      shared  buff/cache   available
    Mem:       16276472      259748     7036492       59720     8980232     8502948
    Swap:             0           0           0

I am expecting the buff/cache value to go quite low, certainly much lower than about 9GB.

I only have the following non [system] processes running:

ps -ef | grep -v \\[

UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 Jul23 ?        00:03:52 /usr/lib/systemd/systemd --switched-root --system --deserialize 24
dbus      1136     1  0 Jul23 ?        00:06:30 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root      1245     1  0 Jul23 ?        00:00:01 /usr/sbin/sshd -D -oCiphers=aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc,3des-cbc -oMACs=umac-128-etm@openssh.com,hmac-sha1-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,umac-128@openssh.com,hmac-sha1,hmac-sha2-256,hmac-sha2-512 -oGSSAPIKexAlgorithms=gss-gex-sha1-,gss-group14-sha1- -oKexAlgorithms=curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1
root      1395     1  0 Jul23 ?        00:02:02 /usr/lib/systemd/systemd-logind
root      1396     1  0 Jul23 ?        00:00:55 /usr/libexec/accounts-daemon
root      1609  1607  0 21:39 ?        00:00:00 sshd: root@pts/5
root      1610  1609  0 21:39 pts/5    00:00:00 -zsh
root      1642  1640  0 21:39 ?        00:00:00 sshd: root@pts/0
root      1643  1642  0 21:39 pts/0    00:00:01 -zsh
root      1668  1665  0 21:40 ?        00:00:00 sshd: root@pts/2
root      1669  1668  0 21:40 pts/2    00:00:00 -zsh
root      1719     1  0 Jul23 tty1     00:00:00 /sbin/agetty -o -p -- \u --noclear tty1 linux
root      2601     1  0 21:47 ?        00:00:00 /usr/lib/systemd/systemd-udevd
root      2685  1669  0 21:55 pts/2    00:00:00 ps -ef
root     13104 13025  0 18:05 ?        00:00:00 sshd: root@pts/1
root     14359     1  0 Jul23 ?        00:00:00 /usr/lib/systemd/systemd --user
root     14360 14359  0 Jul23 ?        00:00:00 (sd-pam)
root     28017 13104  0 21:28 pts/1    00:00:00 -zsh
root     28050 28017  0 21:28 pts/1    00:00:03 top

and top sorted by RAM usage shows:

top - 22:01:59 up 2 days, 13:59,  4 users,  load average: 0.00, 0.00, 0.02
Tasks: 125 total,   1 running,  69 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  0.1 sy,  0.0 ni, 99.8 id,  0.0 wa,  0.0 hi,  0.1 si,  0.0 st
KiB Mem : 16276472 total,  7033768 free,   258980 used,  8983724 buff/cache
KiB Swap:        0 total,        0 free,        0 used.  8502072 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                        
 1640 root      20   0  148024   8320   7216 S   0.0  0.1   0:00.05 sshd                                                                           
13025 root      20   0  148024   8312   7212 S   0.0  0.1   0:00.01 sshd                                                                           
 1665 root      20   0  148024   8220   7116 S   0.0  0.1   0:00.06 sshd                                                                           
 1607 root      20   0  148024   8080   6972 S   0.0  0.0   0:00.03 sshd                                                                           
 2601 root      20   0   97684   7332   5904 S   0.0  0.0   0:00.06 systemd-udevd                                                                  
    1 root      20   0  228444   7884   4944 S   0.0  0.0   3:52.62 systemd                                                                        
 1245 root      20   0   82792   5536   4708 S   0.0  0.0   0:01.97 sshd                                                                           
 1610 root      20   0  164076  11060   4228 S   0.0  0.1   0:00.09 zsh                                                                            
28017 root      20   0  164076  11016   4184 S   0.0  0.1   0:00.04 zsh                                                                            
 1669 root      20   0  168436  11336   4164 S   0.0  0.1   0:00.60 zsh                                                                            
 1643 root      20   0  164076  10928   4016 S   0.0  0.1   0:02.00 zsh    



cat /proc/vmstat

nr_free_pages 1758411
nr_zone_inactive_anon 17180
nr_zone_active_anon 9584
nr_zone_inactive_file 1049
nr_zone_active_file 4853
nr_zone_unevictable 0
nr_zone_write_pending 1
nr_mlock 0
nr_page_table_pages 1036
nr_kernel_stack 2080
nr_bounce 0
nr_zspages 0
nr_free_cma 0
numa_hit 3000299470
numa_miss 0
numa_foreign 0
numa_interleave 21375
numa_local 3000299470
numa_other 0
nr_inactive_anon 17180
nr_active_anon 9584
nr_inactive_file 1049
nr_active_file 4853
nr_unevictable 0
nr_slab_reclaimable 427423
nr_slab_unreclaimable 1797722
nr_isolated_anon 0
nr_isolated_file 0
workingset_refault 19330098
workingset_activate 3316794
workingset_nodereclaim 1045284
nr_anon_pages 11842
nr_mapped 4365
nr_file_pages 20844
nr_dirty 5
nr_writeback 0
nr_writeback_temp 0
nr_shmem 14930
nr_shmem_hugepages 0
nr_shmem_pmdmapped 0
nr_anon_transparent_hugepages 0
nr_unstable 0
nr_vmscan_write 526329
nr_vmscan_immediate_reclaim 47959
nr_dirtied 90587620
nr_written 90117830
nr_dirty_threshold 344348
nr_dirty_background_threshold 171963
pgpgin 329849668
pgpgout 656116433
pswpin 311639
pswpout 526807
pgalloc_dma 12
pgalloc_dma32 473350531
pgalloc_normal 2600312635
pgalloc_movable 0
allocstall_dma 0
allocstall_dma32 0
allocstall_normal 2594
allocstall_movable 7924
pgskip_dma 0
pgskip_dma32 0
pgskip_normal 0
pgskip_movable 0
pgfree 3087941289
pgactivate 31241868
pgdeactivate 54808523
pglazyfree 165199
pgfault 3727210961
pgmajfault 376546
pglazyfreed 88855
pgrefill 56262502
pgsteal_kswapd 93158250
pgsteal_direct 5218459
pgscan_kswapd 94863466
pgscan_direct 5274174
pgscan_direct_throttle 0
zone_reclaim_failed 0
pginodesteal 1983292
slabs_scanned 150018640
kswapd_inodesteal 13603030
kswapd_low_wmark_hit_quickly 37060
kswapd_high_wmark_hit_quickly 1617
pageoutrun 91741
pgrotated 582111
drop_pagecache 82
drop_slab 82
oom_kill 0
numa_pte_updates 0
numa_huge_pte_updates 1
numa_hint_faults 0
numa_hint_faults_local 0
numa_pages_migrated 0
pgmigrate_success 12440716
pgmigrate_fail 2270900
compact_migrate_scanned 356381883
compact_free_scanned 1233511266
compact_isolated 27255306
compact_stall 1024
compact_fail 793
compact_success 231
compact_daemon_wake 75068
compact_daemon_migrate_scanned 315527488
compact_daemon_free_scanned 936465547
htlb_buddy_alloc_success 0
htlb_buddy_alloc_fail 0
unevictable_pgs_culled 11829
unevictable_pgs_scanned 0
unevictable_pgs_rescued 14
unevictable_pgs_mlocked 14
unevictable_pgs_munlocked 8
unevictable_pgs_cleared 6
unevictable_pgs_stranded 0
thp_fault_alloc 6978
thp_fault_fallback 327
thp_collapse_alloc 405
thp_collapse_alloc_failed 3405
thp_file_alloc 0
thp_file_mapped 0
thp_split_page 0
thp_split_page_failed 0
thp_deferred_split_page 7280
thp_split_pmd 1
thp_split_pud 0
thp_zero_page_alloc 0
thp_zero_page_alloc_failed 0
thp_swpout 1
thp_swpout_fallback 0
balloon_inflate 0
balloon_deflate 0
balloon_migrate 0
swap_ra 184300
swap_ra_hit 166308

I've even tried:

echo 1000 >  /proc/sys/vm/vfs_cache_pressure

and

echo 5000000 >  /proc/sys/vm/min_free_kbytes

but even then, after echo 3 > /proc/sys/vm/drop_caches which does nothing to the cache still leaving about 9GB of cache in use

What is using all my cache and why can't I get rid of it?

I left the system like the above (next to nothing running, cache won't clear manually) for about 10 hours (overnight) and sometime during that period the cache dropped down to about the size that it is immediately after reboot.

total used free shared buff/cache available Mem: 16276472 245420 15454540 59704 576512 14101780

This still leaves me with the original problem - What is using all my cache and why can't I get rid of it?

closed as unclear what you're asking by Kiwy, schily, Jesse_b, slm Jul 25 '18 at 17:37

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • why do you want to clear cache and buffer ? OS is handeling it better than you will .Do not bother except if you have the bests of the reasons which I highly doubt. Whitout a good reason to clear cache, your question is unclear and unjustified, OS will release cache when necessary. – Kiwy Jul 25 '18 at 14:12
  • @Kiwy - he doesn't need/want to clear it, he's trying to understand why so much is still in use with a minimal number of processes running. Of course one of them is systemd so if this were /. or similar I'd expect 1001 hate replies about that being the cause. Anyway, I think it is a good question, although I didn't see it before the recent edit. – ivanivan Jul 25 '18 at 22:11
  • any mapped (mmap()) file will pin the cache. – sourcejedi Jul 25 '18 at 22:17
  • To enumerate mapped files and their sizes, run sudo du -aLh /proc/*/map_files/ | sort -h – sourcejedi Jul 25 '18 at 22:19
  • But, that's not your case. The smoking gun is nr_slab_unreclaimable 1797722. 1797722 * 4096 (PAGE_SIZE) = 7363469312 i.e. 7.3GB. You need to run slabtop. (And since it's "unreclaimable", you can't just ask to drop it like a cache, you need to find what it is and stop it...). – sourcejedi Jul 25 '18 at 22:26

Browse other questions tagged or ask your own question.