1

The box has 96GB memory, no swap.

# free -m
             total       used       free     shared    buffers     cached
Mem:         96714      85762      10952          0         94       2185
-/+ buffers/cache:      83482      13232
Swap:            0          0          0

It looks like 85GB memory are in use, but from top command, sort by %MEM

Only 32GB memory are in use, where did the rest go?

enter image description here

Any ideas?

EDIT

# cat /proc/meminfo
MemTotal:         99036140     kB
MemFree:          10901516     kB
Buffers:          126816       kB
Cached:           2355968      kB
SwapCached:       0            kB
Active:           35103352     kB
Inactive:         2973732      kB
Active(anon):     34352040     kB
Inactive(anon):   1793248      kB
Active(file):     751312       kB
Inactive(file):   1180484      kB
Unevictable:      1892         kB
Mlocked:          0            kB
SwapTotal:        0            kB
SwapFree:         0            kB
Dirty:            8864         kB
Readahead:        0            kB
Writeback:        0            kB
AnonPages:        35596308     kB
Mapped:           693268       kB
Shmem:            549000       kB
Slab:             48846688     kB
SReclaimable:     48771520     kB
SUnreclaim:       75168        kB
KernelStack:      9864         kB
PageTables:       108668       kB
NFS_Unstable:     0            kB
Bounce:           0            kB
WritebackTmp:     0            kB
CommitLimit:      49518068     kB
Committed_AS:     41937368     kB
VmallocTotal:     34359738367  kB
VmallocUsed:      500436       kB
VmallocChunk:     34290219936  kB
HugePages_Total:  0
HugePages_Free:   0
HugePages_Rsvd:   0
HugePages_Surp:   0
Hugepagesize:     2048         kB
DirectMap4k:      7252         kB
DirectMap2M:      100620288    kB
  • did you try running other memory analysis tools, which might help understanding which number holds the correct memory usage? – Yaron May 18 '17 at 6:59
  • @Yaron what tools do you recommend? – daisy May 18 '17 at 7:02
  • which unix flavor are you using? – Yaron May 18 '17 at 7:35
  • @Yaron It's centos 6.X – daisy May 18 '17 at 7:36
  • before running GUI tools, can you please execute cat /proc/meminfo? – Yaron May 18 '17 at 7:39
0

Total Memory = Free Memory + Cache/Buffers + Active + Inactive

         = 10901516 + 126816 + 2355968 + 35103352 + 2973732
         = 51461384 KB
         = 49 GB 

Missing Memory = 96 GB - 49 GB = 47 GB The missing memory almost equal to Slab Memory (48846688 kB ) , I am guessing that there some kernel module is leaking memory.

Please print /proc/slabinfo & slabtop command output to investigate more, and also print the output of vmware-toolbox-cmd stat balloon & vmware-toolbox-cmd stat memlimit if you're running on vmware.

Edit

It look like there is dentry kernel module consuming 48646536kB , searching the internet I found https://serverfault.com/questions/561350/unusually-high-dentry-cache-usage , it suggest that issue come from The culprit was a NSS (Network Security Services) library bundled with Libcurl

You only have to upgrade nss-softokn (which has a required dependency on nss-utils) at a minimum. and to get the benefit, you need to set the environment variable NSS_SDB_USE_CACHE for the process that is using libcurl. the presence of that environment variable is what allows the costly non-existent file checks to be skipped."

Give it try and let us know.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.