6

I got a daily crontab task:

50 1 * * * sh /my_path/daily_task.sh > /tmp/zen_log 2>&1

This daily_task shell script will run some python scripts and produce a data file.

And it fails for two nights. But when I came in the morning, run the python scripts manually, I got the data file. Or I set a new crontab which only set the date to 0 10 * * *, and this crontab succeed, too.

So yesterday, I put > /tmp/zen_log 2>&1 in the cron task to get some error message.

And this morning, I got this error message in zen_log:

/my_path/daily_task.sh: line 19: 12364 Killed /usr/local/bin/python2.7 my_python_script.py 2 mix > mix_hc_$datestamp 2>&1

It seems some process has been killed? But what exactly does this line 19: 12364 Killed mean?


PS:

Today, one minute ago, when I manually run the python script, I got: /usr/local/bin/python2.7 my_python_script.py 2 mix > mix_hc_$datestamp 2>&1 Killed

  • What's line 19 in the script? Maybe posting your script will help us provide you with an answer. – devnull Feb 12 '15 at 3:36
  • line 19 is /usr/local/bin/python2.7 my_python_script.py 2 mix > mix_hc_$datestamp 2>&1 – Zen Feb 12 '15 at 3:37
  • Can you update your question with the contents of daily_task.sh? It's hard to understand why it fails around 1:50 am but succeeds at 10 am with the information so far. – devnull Feb 12 '15 at 3:42
  • 3
    Also check the contents of /var/log/messages I wonder if your script is creating an Out-of-Memory (OOM) error and being killed. Does your system tend to run other system resource hungry scripts/applications/jobs during the late night hours compared to the AM hours? – devnull Feb 12 '15 at 3:48
  • @DevNull, I checked the kernel log, now I'm sure this script has occupied too much memory and the kernel killed it. – Zen Feb 12 '15 at 4:43
15

Often times when applications are being killed, it is always a good idea to take a quick look at your /var/log/messages file to see if the kernel is killing the process. The most common trigger (in my experience) has always been due to out-of-memory (OOM) errors, since my company primarily uses java applications, it is quite common for the devs to publish a bad code update that triggers an OOM event.

Scheduling tasks when your OS has the most available resources is probably why it is succeeding in the AM time slots and not in the PM when most people like to schedule taxing system jobs. Simple solutions to this are to either increase your systems resources, restrict resources allocated to your code or move around when your jobs are scheduled so they don't conflict.

  • Just extra note that by default ubuntu output the messages into syslog instead of /var/log/messages – oak Jun 12 '18 at 10:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.