-1

My CentOS server crashed with 100% memory with swapping, after running fine for a week. Investigating the /var/log/messages shows me the following: https://pastebin.com/CSF8DDzX Basically, alot of the following:

Feb 12 23:31:32 [localhost] kernel: [31186]     0 31186    85296        0     116     1108             0 php
Feb 12 23:31:32 [localhost] kernel: [31252]     0 31252    45588        1      44      230             0 crond
Feb 12 23:31:32 [localhost] kernel: [31253]     0 31253    85296        0     112     1108             0 php
Feb 12 23:31:32 [localhost] kernel: [31315]     0 31315    45588        1      44      230             0 crond
Feb 12 23:31:32 [localhost] kernel: [31316]     0 31316    85296        0     116     1107             0 php
Feb 12 23:31:32 [localhost] kernel: [31377]     0 31377    45588        1      44      230             0 crond

It looks like it is stuck in a loop. My crontab -e shows me this:

0 0,4,8,12,16,20 * * * php PeriodicallyKillWorkers.php
* * * * * php cheduleWorker.php

The scripts work fine, as does/did crontab. Where do I investigate further?

  • Please if you downvote my question, tell me why so I can improve it. – Gudgip Feb 14 at 10:11
1

You run this every minute:

* * * * * php cheduleWorker.php

Check what is inside and how much time it take to exec this script. And add full paths to the program and script

Also check the other job (how long in run, how much memory it use and so on)

  • if I time this php script: real 0m0.053s What it does is loop over a few database records and publish a rabbitmq message. I don't think the real paths will give you any additional info. What might be important is that the files are located on an network drive which disconnected for a small period of time. – Gudgip Feb 14 at 10:09
  • 1
    @Gudgip, this (network drive) can cause the problem. Better wrap the php in shell checking if drive is mounted. Also check the other job (how long in run, how much memory it use and so on) – Romeo Ninov Feb 14 at 10:14
  • The other job time: real 0m0.260s. This job kills all supervisord (=rabbitmq worker) jobs every 4 hours. Wrapping in a shell is indeed a good idea. Do you have any idea why it produces so much memory? – Gudgip Feb 14 at 10:17
  • @Gudgip, it can be some deadlock when you deal with MQ/database. Better add to the script a lot of control prints and via shell script add dedicated for each cron instance log and check. – Romeo Ninov Feb 14 at 10:32
  • Seems reasonable. I'll try this. :) – Gudgip Feb 14 at 12:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.