What does `line 19: 12364 Killed` mean in crontab error message?

Question

I got a daily crontab task:

50 1 * * * sh /my_path/daily_task.sh > /tmp/zen_log 2>&1

This daily_task shell script will run some python scripts and produce a data file.

And it fails for two nights. But when I came in the morning, run the python scripts manually, I got the data file. Or I set a new crontab which only set the date to 0 10 * * *, and this crontab succeed, too.

So yesterday, I put > /tmp/zen_log 2>&1 in the cron task to get some error message.

And this morning, I got this error message in zen_log:

/my_path/daily_task.sh: line 19: 12364 Killed /usr/local/bin/python2.7 my_python_script.py 2 mix > mix_hc_$datestamp 2>&1

It seems some process has been killed? But what exactly does this line 19: 12364 Killed mean?

---

PS:

Today, one minute ago, when I manually run the python script, I got: /usr/local/bin/python2.7 my_python_script.py 2 mix > mix_hc_$datestamp 2>&1 Killed

Answer 1

Often times when applications are being killed, it is always a good idea to take a quick look at your /var/log/messages file to see if the kernel is killing the process. The most common trigger (in my experience) has always been due to out-of-memory (OOM) errors, since my company primarily uses java applications, it is quite common for the devs to publish a bad code update that triggers an OOM event.

Scheduling tasks when your OS has the most available resources is probably why it is succeeding in the AM time slots and not in the PM when most people like to schedule taxing system jobs. Simple solutions to this are to either increase your systems resources, restrict resources allocated to your code or move around when your jobs are scheduled so they don't conflict.