3

I have a production machine running a 2.6 kernel (Centos 6.7) with 64 GB of RAM of which 18 GB is apparently used for buffer memory.

I am interested in identifying specifically what that 18 GB is being used for so I can tell whether it is pathological or not.

This machine is a busy front-end load balancer + web server so it has a good deal of incoming as well as internal TCP connections. It does a fair bit of small disk I/Os randomly into a large (10's of millions of files) directory tree. It also writes a lot of data to web server log files, maybe a few 10's of GB per day.

I have tried using slabtop but the size of the slabs only seems to account for a portion of this usage (about 8 or 9 GB at the time of this writing.)

Is there any way to get a clearer picture of what constitutes the 18 GB so I can tell whether this is reasonable or not?

Here are some basic diagnostics:

uname -a:

Linux [HOSTNAME REMOVED] 2.6.32-431.11.2.el6.x86_64 #1 SMP Tue Mar 25 19:59:55 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

free -m:

             total       used       free     shared    buffers     cached
Mem:         64377      64099        278         49      17935      34717
-/+ buffers/cache:      11446      52931
Swap:        32767        980      31787

/proc/meminfo:

MemTotal:       65922808 kB
MemFree:          257868 kB
Buffers:        18353708 kB
Cached:         35536088 kB
SwapCached:        47708 kB
Active:         28975120 kB
Inactive:       27143572 kB
Active(anon):    1514004 kB
Inactive(anon):   765492 kB
Active(file):   27461116 kB
Inactive(file): 26378080 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:      33554424 kB
SwapFree:       32550900 kB
Dirty:             15792 kB
Writeback:             0 kB
AnonPages:       2192080 kB
Mapped:            73996 kB
Shmem:             50516 kB
Slab:            8880128 kB
SReclaimable:    8392424 kB
SUnreclaim:       487704 kB
KernelStack:        8376 kB
PageTables:       139560 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    66515828 kB
Committed_AS:    6578468 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      393776 kB
VmallocChunk:   34325005944 kB
HardwareCorrupted:     0 kB
AnonHugePages:    167936 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        5056 kB
DirectMap2M:     2045952 kB
DirectMap1G:    65011712 kB

slabtop -sc:

 Active / Total Objects (% used)    : 24709382 / 28507117 (86.7%)
 Active / Total Slabs (% used)      : 2211714 / 2211815 (100.0%)
 Active / Total Caches (% used)     : 107 / 183 (58.5%)
 Active / Total Size (% used)       : 8026764.94K / 8620046.54K (93.1%)
 Minimum / Average / Maximum Object : 0.02K / 0.30K / 4096.00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
5479168 5196414  94%    0.99K 1369792        4   5479168K ext4_inode_cache
6662980 6648409  99%    0.19K 333149       20   1332596K dentry
11147878 9956546  89%    0.10K 301294       37   1205176K buffer_head
626199 573720  91%    0.55K  89457        7    357828K radix_tree_node
3458639 1395404  40%    0.06K  58621       59    234484K size-64
309529 278095  89%    0.20K  16291       19     65164K vm_area_struct
 94365  83090  88%    0.25K   6291       15     25164K tw_sock_TCP
 74940  65974  88%    0.30K   6245       12     24980K nf_conntrack_ffffffff81b18540
  8668   8054  92%    1.69K   2167        4     17336K TCP
 98520  87434  88%    0.12K   3284       30     13136K size-128
   182    182 100%   32.12K    182        1     11648K kmem_cache
 10132   9261  91%    1.00K   2533        4     10132K size-1024
 49120  48320  98%    0.19K   2456       20      9824K size-192
157003 116353  74%    0.05K   2039       77      8156K anon_vma_chain
 10170   9082  89%    0.69K   2034        5      8136K sock_inode_cache
 20090  13010  64%    0.38K   2009       10      8036K ip_dst_cache
 28620  18268  63%    0.19K   1431       20      5724K filp
  5856   5069  86%    0.64K    976        6      3904K proc_inode_cache
   918    911  99%    4.00K    918        1      3672K size-4096
 76176  45154  59%    0.04K    828       92      3312K anon_vma
  4055   3986  98%    0.77K    811        5      3244K shmem_inode_cache
  4686   4500  96%    0.58K    781        6      3124K inode_cache
 20979  20965  99%    0.14K    777       27      3108K sysfs_dir_cache
  1155   1053  91%    2.59K    385        3      3080K task_struct
  5082   4059  79%    0.50K    726        7      2904K skbuff_fclone_cache
  1396   1331  95%    2.00K    698        2      2792K size-2048
  1029    927  90%    2.06K    343        3      2744K sighand_cache
    35     35 100%   64.00K     35        1      2240K size-65536
   139    108  77%   16.00K    139        1      2224K size-16384
  3760   3566  94%    0.50K    470        8      1880K size-512
  8660   4042  46%    0.19K    433       20      1732K cred_jar
 24898  22924  92%    0.06K    422       59      1688K tcp_bind_bucket
 12030   9242  76%    0.12K    401       30      1604K eventpoll_epi
   166    143  86%    8.00K    166        1      1328K size-8192
  1162    927  79%    1.06K    166        7      1328K signal_cache
 32592  28332  86%    0.03K    291      112      1164K size-32
  3945   2357  59%    0.25K    263       15      1052K skbuff_head_cache
   640    465  72%    1.38K    128        5      1024K mm_struct
   254    254 100%    4.00K    254        1      1016K names_cache
 12243   9382  76%    0.07K    231       53       924K eventpoll_pwq
     7      7 100%  128.00K      7        1       896K size-131072
   990    563  56%    0.81K    110        9       880K task_xstate
  9858   9061  91%    0.07K    186       53       744K selinux_inode_security
   836    496  59%    0.69K     76       11       608K files_cache
   594    453  76%    0.81K     66        9       528K UDP
....

iostat -x -m 10:

Linux 2.6.32-431.11.2.el6.x86_64 ([HOSTNAME REMOVED])  11/25/2015      _x86_64_        (32 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          29.29    0.04    5.51    1.52    0.00   63.64

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00    0.00    0.00     0.00     0.00    43.76     0.00    5.52   5.05   0.00
sdb              57.38   377.29  113.86   91.39     2.82     1.83    46.36     0.04    0.19   0.10   1.99

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          44.62    0.00    8.82    0.59    0.00   45.98

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb               3.50   664.60    8.10  174.90     0.14     3.28    38.28     2.70   14.78   1.37  25.09

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          34.30    0.00    8.37    0.14    0.00   57.19

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb               0.10   377.70    8.90   48.60     0.21     1.67    66.71     0.24    4.21   2.19  12.62
1
3

Buffers field of free is representing all the I/O directly to block devices on your host, either via direct-IO or via filesystem metadata blocks ex: find etc. You can find more info on which devices using these pages look at my blog page here

slabtop is representing separate cache area used by kernel and it's allocated for frequently used kernel objects. Looking at your slabinfo output, there is about ~7GB of ext4 inode and dentry cache. I suspect their corresponding disk blocks must be the one's showing up in 'Buffers' field

1
  • I'm marking this one as the answer because it points out important details: (1) The breakdown of what "filesystem metadata" is specifically -- direct IO AND filesystem metadata blocks; (2) that slab is separate from buffers and cache; (3) that entries for inode/dentry that are in slabs will have corresponding disk blocks in as part of buffer. For more details on slab being separate from buffers/cache see stackoverflow.com/a/5467207/4481063 – mutron Dec 2 '15 at 0:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.