14

I got a daily crontab task:

50 1 * * * sh /my_path/daily_task.sh > /tmp/zen_log 2>&1

This daily_task shell script will run some python scripts and produce a data file.

And it fails for two nights. But when I came in the morning, run the python scripts manually, I got the data file. Or I set a new crontab which only set the date to 0 10 * * *, and this crontab succeed, too.

So yesterday, I put > /tmp/zen_log 2>&1 in the cron task to get some error message.

And this morning, I got this error message in zen_log:

/my_path/daily_task.sh: line 19: 12364 Killed /usr/local/bin/python2.7 my_python_script.py 2 mix > mix_hc_$datestamp 2>&1

It seems some process has been killed? But what exactly does this line 19: 12364 Killed mean?

PS:

Today, one minute ago, when I manually run the python script, I got: /usr/local/bin/python2.7 my_python_script.py 2 mix > mix_hc_$datestamp 2>&1 Killed

Improve this question
7
  • What's line 19 in the script? Maybe posting your script will help us provide you with an answer.
    – devnull
    Feb 12, 2015 at 3:36
  • line 19 is /usr/local/bin/python2.7 my_python_script.py 2 mix > mix_hc_$datestamp 2>&1
    – Zen
    Feb 12, 2015 at 3:37
  • Can you update your question with the contents of daily_task.sh? It's hard to understand why it fails around 1:50 am but succeeds at 10 am with the information so far.
    – devnull
    Feb 12, 2015 at 3:42
  • 3
    Also check the contents of /var/log/messages I wonder if your script is creating an Out-of-Memory (OOM) error and being killed. Does your system tend to run other system resource hungry scripts/applications/jobs during the late night hours compared to the AM hours?
    – devnull
    Feb 12, 2015 at 3:48
  • @DevNull, I checked the kernel log, now I'm sure this script has occupied too much memory and the kernel killed it.
    – Zen
    Feb 12, 2015 at 4:43

1 Answer 1

Reset to default
22

Often times when applications are being killed, it is always a good idea to take a quick look at your /var/log/messages file to see if the kernel is killing the process. The most common trigger (in my experience) has always been due to out-of-memory (OOM) errors, since my company primarily uses java applications, it is quite common for the devs to publish a bad code update that triggers an OOM event.

Scheduling tasks when your OS has the most available resources is probably why it is succeeding in the AM time slots and not in the PM when most people like to schedule taxing system jobs. Simple solutions to this are to either increase your systems resources, restrict resources allocated to your code or move around when your jobs are scheduled so they don't conflict.

Improve this answer
1
  • 1
    Just extra note that by default ubuntu output the messages into syslog instead of /var/log/messages
    – oak
    Jun 12, 2018 at 10:15

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .