2

I have a dedicated MySQL server equipped with 128 GB RAM. MySQL recently gets killed by the oom-killer, although MySQL is configured to use 95 GB in the worst case. In my research I came across this:

# cat /proc/11895/status
Name:   mysqld
State:  S (sleeping)
Tgid:   11895
Pid:    11895
PPid:   24530
TracerPid:      0
Uid:    27      27      27      27
Gid:    27      27      27      27
Utrace: 0
FDSize: 1024
Groups: 27
VmPeak: 72188044 kB
VmSize: 72122508 kB
VmLck:         0 kB
VmHWM:  33294036 kB
VmRSS:  32829668 kB
VmData: 72076496 kB
VmStk:        88 kB
VmExe:     11800 kB
VmLib:      3608 kB
VmPTE:     73388 kB
VmSwap:  4139376 kB
Threads:        59

I'm wondering, why is the VmHWM and VmRSS at only around 33 GB whereas on another server (also a slave to the same master, configured almost the same (except for buffer pool), except, that it has 256 GB RAM), the output is as follows:

# cat /proc/51298/status
Name:   mysqld
State:  S (sleeping)
Tgid:   51298
Pid:    51298
PPid:   50443
TracerPid:      0
Uid:    27      27      27      27
Gid:    27      27      27      27
Utrace: 0
FDSize: 2048
Groups: 27
VmPeak: 243701128 kB
VmSize: 239628932 kB
VmLck:         0 kB
VmHWM:  209331200 kB
VmRSS:  205515868 kB
VmData: 239582156 kB
VmStk:        88 kB
VmExe:     11800 kB
VmLib:      3608 kB
VmPTE:    409600 kB
VmSwap:        0 kB
Threads:        281

Here the memory is used to about 80%, whereas on the oom-killed server it's only about 25% (note, that these values where observed shortly before the oom-killer strikes again). What could be the reason? There is no competing process. And what can I do about it?

1 Answer 1

1

So, it turns out a colleague was experimenting with large-page-support and didn't revert all changes he made. When I ran

sysctl -w vm.nr_hugepages=0

and commented out this section in the /etc/sysctl.conf

# Hugepage Support MySQL
#vm.hugetlb_shm_group = 27
#kernel.shmmax = 10737418240
#kernel.shmall = 23689185
#vm.nr_hugepages = 46268

it freed up 90 GB which were wasted. This could be seen in the output of cat /proc/meminfo:

HugePages_Total:   46268
HugePages_Free:    46268
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

Huge thanks go out to Matthew Ife. Please upvote his answer over at serverfault.com instead of this one.

1
  • note: shmmax and shmall aren't related to hugepages.
    – Massimo
    Commented Aug 27, 2018 at 20:29

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .