Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I'm running Ubuntu 14.04 on Digitalocean to host my blog. I'm using HHVM + nginx + MySQL. Every few hours (~12h), there's no more RAM left and nginx throws an 502 error.

I check the memory usage for the last 10 minutes with free -m and that's what I get:

             total       used       free     shared    buffers     cached
Mem:           994        714        279         29         20        235
-/+ buffers/cache:        458        535
Swap:            0          0          0

             total       used       free     shared    buffers     cached
Mem:           994        715        278         29         20        235
-/+ buffers/cache:        459        534
Swap:            0          0          0

             total       used       free     shared    buffers     cached
Mem:           994        722        271         29         20        240
-/+ buffers/cache:        461        532
Swap:            0          0          0

             total       used       free     shared    buffers     cached
Mem:           994        729        264         29         20        240
-/+ buffers/cache:        469        524
Swap:            0          0          0

             total       used       free     shared    buffers     cached
Mem:           994        725        268         29         20        240
-/+ buffers/cache:        464        529
Swap:            0          0          0

So how can I figure out where the memory leak comes from?

If it helps, the htop output:

  PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
 2107 root       20   0 26284  2528  1408 R  0.7  0.2  0:00.47 htop
 1002 mysql      20   0  737M 72680  7332 S  0.0  7.1  0:00.99 /usr/sbin/mysqld
 1088 redis      20   0 36996  7256   876 S  0.0  0.7  0:06.46 /usr/bin/redis-server 127.0.0.1:6379
  925 mysql      20   0  737M 72680  7332 S  0.0  7.1  0:17.87 /usr/sbin/mysqld
  949 mysql      20   0  737M 72680  7332 S  0.0  7.1  0:00.21 /usr/sbin/mysqld
    1 root       20   0 33492  2768  1400 S  0.0  0.3  0:01.30 /sbin/init
  343 root       20   0 19608   648   460 S  0.0  0.1  0:00.12 upstart-udev-bridge --daemon
  351 messagebu  20   0 39228  1292   884 S  0.0  0.1  0:00.07 dbus-daemon --system --fork
  354 root       20   0 51476  1552   952 S  0.0  0.2  0:00.05 /lib/systemd/systemd-udevd --daemon
  399 root       20   0 43452  1756  1396 S  0.0  0.2  0:00.00 /lib/systemd/systemd-logind
  427 syslog     20   0  249M  1388   788 S  0.0  0.1  0:00.01 rsyslogd
  428 syslog     20   0  249M  1388   788 S  0.0  0.1  0:00.00 rsyslogd
  429 syslog     20   0  249M  1388   788 S  0.0  0.1  0:00.02 rsyslogd
  426 syslog     20   0  249M  1388   788 S  0.0  0.1  0:00.03 rsyslogd
  452 root       20   0 15408   648   320 S  0.0  0.1  0:00.05 upstart-file-bridge --daemon
  672 root       20   0 15392   692   332 S  0.0  0.1  0:00.05 upstart-socket-bridge --daemon
  810 root       20   0 15820   928   768 S  0.0  0.1  0:00.00 /sbin/getty -8 38400 tty4
  816 root       20   0 15820   928   768 S  0.0  0.1  0:00.00 /sbin/getty -8 38400 tty5
  821 root       20   0 15820   932   768 S  0.0  0.1  0:00.00 /sbin/getty -8 38400 tty2
  822 root       20   0 15820   924   768 S  0.0  0.1  0:00.00 /sbin/getty -8 38400 tty3
  825 root       20   0 15820   932   768 S  0.0  0.1  0:00.00 /sbin/getty -8 38400 tty6
  855 root       20   0 61364  3008  2328 S  0.0  0.3  0:00.03 /usr/sbin/sshd -D
  858 root       20   0  4368   656   512 S  0.0  0.1  0:00.00 acpid -c /etc/acpi/events -s /var/run/acpid.socket
  859 root       20   0 23656  1024   768 S  0.0  0.1  0:00.03 cron
  861 daemon     20   0 19140   160     0 S  0.0  0.0  0:00.00 atd
  894 whoopsie   20   0  327M  3584  2424 S  0.0  0.4  0:00.00 whoopsie
  896 whoopsie   20   0  327M  3584  2424 S  0.0  0.4  0:00.00 whoopsie
  873 whoopsie   20   0  327M  3584  2424 S  0.0  0.4  0:00.00 whoopsie
 1170 www-data   20   0 1541M  390M 91108 S  0.0 39.3  0:00.59 /usr/bin/hhvm --config /etc/hhvm/php.ini --config /etc/hhvm/server.ini --user www-data --mode daemo
 1357 www-data   20   0 1541M  390M 91108 S  0.0 39.3  0:23.19 /usr/bin/hhvm --config /etc/hhvm/php.ini --config /etc/hhvm/server.ini --user www-data --mode daemo
 1437 www-data   20   0 1541M  390M 91108 S  0.0 39.3  0:12.09 /usr/bin/hhvm --config /etc/hhvm/php.ini --config /etc/hhvm/server.ini --user www-data --mode daemo
 1438 www-data   20   0 1541M  390M 91108 S  0.0 39.3  0:19.50 /usr/bin/hhvm --config /etc/hhvm/php.ini --config /etc/hhvm/server.ini --user www-data --mode daemo
 1439 www-data   20   0 1541M  390M 91108 S  0.0 39.3  0:14.87 /usr/bin/hhvm --config /etc/hhvm/php.ini --config /etc/hhvm/server.ini --user www-data --mode daemo
 1440 www-data   20   0 1541M  390M 91108 S  0.0 39.3  0:14.93 /usr/bin/hhvm --config /etc/hhvm/php.ini --config /etc/hhvm/server.ini --user www-data --mode daemo
 1441 www-data   20   0 1541M  390M 91108 S  0.0 39.3  0:13.68 /usr/bin/hhvm --config /etc/hhvm/php.ini --config /etc/hhvm/server.ini --user www-data --mode daemo
 1442 www-data   20   0 1541M  390M 91108 S  0.0 39.3  0:11.29 /usr/bin/hhvm --config /etc/hhvm/php.ini --config /etc/hhvm/server.ini --user www-data --mode daemo
 1443 www-data   20   0 1541M  390M 91108 S  0.0 39.3  0:16.50 /usr/bin/hhvm --config /etc/hhvm/php.ini --config /etc/hhvm/server.ini --user www-data --mode daemo
 1444 www-data   20   0 1541M  390M 91108 S  0.0 39.3  0:17.66 /usr/bin/hhvm --config /etc/hhvm/php.ini --config /etc/hhvm/server.ini --user www-data --mode daemo
  916 www-data   20   0 1541M  390M 91108 S  0.0 39.3  2:25.37 /usr/bin/hhvm --config /etc/hhvm/php.ini --config /etc/hhvm/server.ini --user www-data --mode daemo
  945 mysql      20   0  737M 72680  7332 S  0.0  7.1  0:00.18 /usr/sbin/mysqld
  946 mysql      20   0  737M 72680  7332 S  0.0  7.1  0:00.73 /usr/sbin/mysqld
  947 mysql      20   0  737M 72680  7332 S  0.0  7.1  0:00.23 /usr/sbin/mysqld
  948 mysql      20   0  737M 72680  7332 S  0.0  7.1  0:00.22 /usr/sbin/mysqld
  950 mysql      20   0  737M 66360  7332 S  0.0  6.5  0:00.26 /usr/sbin/mysqld
  951 mysql      20   0  737M 66360  7332 S  0.0  6.5  0:00.23 /usr/sbin/mysqld
  952 mysql      20   0  737M 66360  7332 S  0.0  6.5  0:00.21 /usr/sbin/mysqld
  955 mysql      20   0  737M 66360  7332 S  0.0  6.5  0:00.19 /usr/sbin/mysqld
  956 mysql      20   0  737M 66360  7332 S  0.0  6.5  0:00.17 /usr/sbin/mysqld
 1003 mysql      20   0  737M 66360  7332 S  0.7  6.5  0:01.61 /usr/sbin/mysqld
 1004 mysql      20   0  737M 66360  7332 S  0.0  6.5  0:00.10 /usr/sbin/mysqld
 1005 mysql      20   0  737M 66360  7332 S  0.0  6.5  0:00.00 /usr/sbin/mysqld
 1168 mysql      20   0  737M 66360  7332 S  0.0  6.5  0:00.00 /usr/sbin/mysqld
 1173 mysql      20   0  737M 66360  7332 S  0.0  6.5  0:03.08 /usr/sbin/mysqld
 1202 mysql      20   0  737M 66360  7332 S  0.0  6.5  0:03.37 /usr/sbin/mysqld
 1445 mysql      20   0  737M 66360  7332 S  0.0  6.5  0:02.37 /usr/sbin/mysqld
 1446 mysql      20   0  737M 66360  7332 S  0.0  6.5  0:03.37 /usr/sbin/mysqld
  957 memcache   20   0  357M 42368   876 S  0.0  4.2  0:00.96 /usr/bin/memcached -m 64 -p 11211 -u memcache -l 127.0.0.1
  958 memcache   20   0  357M 42368   876 S  0.0  4.2  0:00.97 /usr/bin/memcached -m 64 -p 11211 -u memcache -l 127.0.0.1
  959 memcache   20   0  357M 42368   876 S  0.0  4.2  0:00.91 /usr/bin/memcached -m 64 -p 11211 -u memcache -l 127.0.0.1
  960 memcache   20   0  357M 42368   876 S  0.0  4.2  0:00.94 /usr/bin/memcached -m 64 -p 11211 -u memcache -l 127.0.0.1
  961 memcache   20   0  357M 42368   876 S  0.0  4.2  0:00.00 /usr/bin/memcached -m 64 -p 11211 -u memcache -l 127.0.0.1
  941 memcache   20   0  357M 42368   876 S  0.0  4.2  0:04.53 /usr/bin/memcached -m 64 -p 11211 -u memcache -l 127.0.0.1
  962 root       20   0 90268  1440   228 S  0.0  0.1  0:00.00 nginx: master process /usr/sbin/nginx
  963 www-data   20   0 91172  2840  1072 S  0.0  0.3  0:01.47 nginx: worker process
 1064 root       20   0 25344  1556  1276 S  0.0  0.2  0:00.04 /usr/lib/postfix/master
 1069 postfix    20   0 27460  1516  1232 S  0.0  0.1  0:00.00 qmgr -l -t unix -u
 1091 redis      20   0 36996  7256   876 S  0.0  0.7  0:00.00 /usr/bin/redis-server 127.0.0.1:6379
 1092 redis      20   0 36996  7256   876 S  0.0  0.7  0:00.00 /usr/bin/redis-server 127.0.0.1:6379
 1151 root       20   0 15820   932   768 S  0.0  0.1  0:00.01 /sbin/getty -8 38400 tty1
 1859 postfix    20   0 27408  1488  1212 S  0.0  0.1  0:00.00 pickup -l -t unix -u -c
 1984 root       20   0  103M  4220  3224 S  0.0  0.4  0:00.11 sshd: root@pts/0
 2054 root       20   0 22764  3876  1676 S  0.0  0.4  0:00.05 -bash
share|improve this question

closed as unclear what you're asking by Patrick, slm, msw, vonbrand, Chris Down Jun 16 at 8:15

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question.If this question can be reworded to fit the rules in the help center, please edit the question.

    
Please read linuxatemyram.com. If you want help troubleshooting your nginx issue, you should provide information on that (logs, error message, system memory profile at the time of the issue, etc). –  Patrick Jun 15 at 21:14
    
valgrind is the usual (free) tool to use to check for memory leaks. –  Faheem Mitha Jun 15 at 21:51
2  
The free -m fluctuations are little and you cannot associate them with a single process, even less only with those processes and not with others. Try to see if this page can help you. –  Hastur Jun 15 at 22:28
    
Also you should not disable completely the swap it is sometime useful –  Kiwy Jun 16 at 9:33
add comment

1 Answer 1

up vote 1 down vote accepted

Nothing to see with what you have provided of information. Your free -m just shows that you have only used around half of your memory. Have you checked your HHVM and nginx logs to see if this is the actual problem?

You could make a small script like this that will take the output of ps aux and append it to a file every 30 second (or more). Let it run in the background in a period and analyse the memory output after. Various UNIX tools can be used on the file with the output afterwards for great analysis opportunities i think.

The script could look like this:

while [ true ]
do
  printf "\n\n" >> /root/psaux_analysis
  echo "--------------------------------------" >> /root/psaux_analysis
  date >> /root/psaux_analysis
  echo "--------------------------------------" >> /root/psaux_analysis
  printf "\n" >> /root/psaux_analysis
  ps aux >> /root/psaux_analysis
  sleep 30;
done

This will give you output similar to the following:

--------------------------------------
Mon Jun 16 07:44:03 UTC 2014
--------------------------------------

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0   2028   584 ?        Ss   Mar12   1:08 init [2]      
root         2  0.0  0.0      0     0 ?        S    Mar12   0:00 [kthreadd/146]
root         3  0.0  0.0      0     0 ?        S    Mar12   0:00 [khelper/146]
daemon     313  0.0  0.0   1804   512 ?        Ss   Mar12   0:00 /sbin/portmap
daemon     528  0.0  0.0   2156   304 ?        Ss   Mar12   0:00 /usr/sbin/atd
104        551  0.0  0.0   2580   240 ?        Ss   Mar12   0:00 /usr/bin/dbus-daemon --system
root       560  0.0  0.0   2288   732 ?        Ss   Mar12   0:28 /usr/sbin/cron
root       727  0.0  0.0  29576  4128 ?        Sl   Mar12  38:12 /usr/bin/python /usr/bin/fail2ban-server -b -s /var/run/fail2ban/fail2ban.sock
...
share|improve this answer
    
You're right, half of memory is free at this time. But you can see, that memory usage gets higher from minute to minute. Nginx logs doesn't help. There are just the upstream errors when the website is already down. And I'm using HHVM and not php-fpm, but the error logs are clear. Maybe I have to increase the log level. –  Slevin Jun 15 at 21:14
    
Your memory in M goes 535 -> 534 -> 532 -> 524 -> 529. So jumps up again in the last one? It seems like. This is the number from the free row in buffer/cache line, which is the interesting one. I would like to hear if you found out what was going on. Good idea with the log level, if possible. –  Adionditsak Jun 16 at 7:05
    
After a couple of hours, it's at 338. I checked the logs again. HHVM logs is at 'warning' level, high enough. There are some code warnings, but nothing critical that should use the memory so much. I think I should use valgrind the figure out, where the leak comes from. –  Slevin Jun 16 at 7:12
    
I have expanded my first answer. I think that can help out a bit :-) –  Adionditsak Jun 16 at 7:40
    
Thanks! I will run it for an hour and post the results on pastebin. –  Slevin Jun 16 at 8:11
show 2 more comments

Not the answer you're looking for? Browse other questions tagged or ask your own question.