Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Usually when a cron job crash, it will leave some error messages in the log.

We run shell script and some java program with cron job. Recently we found some weird thing out from the log. Obviously the program was either crashed or killed because there is a program lock we set when the program was initialed, which was not released. We guess the program was killed because the log of the program didn't show the finish message.

Who can possibly killed the job and how can I get notified through email when a cron job is dead?

EDIT: I don't want the crontab way to receive email because it just push every standard output to the email. In my case there are a lot of other system output from different program because some of them ain't using log4j or they are echo by shell script. Because there are many users in the system, we can't require all the users to manage their program's standard output.

share|improve this question

2 Answers 2

To debug this you can put

set -e -u

at the top of your shell script - it then ends with an error exit status when a command fails or an undefined variable is used.

Then you can call from the cron-job a wrapper script that calls the main script like this

sh -x main_script.sh || echo Failed with exit status: $?

With -x every line is printed out before it is executed. The output is mailed by the cron daemon to you.

You can also use a temp file when the output is too big:

sh -x main_script.sh > $TEMPFILE 2>&1
if [ $? -ne 0 ]; then echo Failed with exit status $? - see $TEMPFILE; fi

In case the exit status is > 128 the command was interrupted by a signal - e.g. someone 'killed' it, a segmentation fault occured or there was a out-of-memory situtation (how to get the signal from the exit status).

share|improve this answer

Check your system logs. Which logs to check for depend on your installation; on Debian with the default setup, you get:

  • in /var/log/auth.log, notices from when the cron job started and finished, because the job involved a PAM session;
  • in /var/log/syslog, a notice that grandchild #32283 failed with exit status 1.
  • an additional notice in /var/log/kern.log if your process was terminated by the OOM killer.

You will get email from cron if your cron job produces any output on its standard output or standard error (unless your local mail delivery system isn't set up properly). You won't get a mail if it quietly returns a nonzero status (which includes the case of being killed by a signal). If you want a notice, arrange a shell wrapper that's noisy in case of error, e.g.

42 1 * * * /path/to/real/job || echo $?

If you want to log more information about processes and how they die (and how they're born, but here you already know), see Is there a log of past threads that are now closed?

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.