0

I have several cronjobs which runs at different times in a day, but one particular cron job is not running as expected and getting terminated after sometime.

0 0  * * * python3 /scratch/pyscripts/backdoor.py --user SEKHAR >> /scratch/tlog/backdoor.log 2>&1;

backdoor.py script will execute each file one by one in for loop, it is abruptly terminating after 1 hours or around executing 25 files as so. There is neither error message in the log file nor the exit message.

But when it is executing the manually, it is running smooth.

How to debug the why this particular cronjob is failing ?

OS : linux-debian

0

2 Answers 2

1

I have cron jobs that can last several hours, so I don't think it's anything inherent in cron that's limiting your task. My inclination is that it's your python task itself that's crashing out (but I do appreciate I've no idea what it's doing or how it's been written, and I do see that you say it runs correctly from a terminal session).

I would probably approach the problem of identifying the root cause of unexpected termination by creating a wrapper around the python job itself. Something like this,

#!/bin/sh
#
exec 1>/scratch/tlog/backdoor.log 2>&1

dtStart=$(date +'%Y-%m-%d %H:%M')
printf "%s\tStarted at %s\n" "$dtStart" "$dtStart"

python3 /scratch/pyscripts/backdoor.py --user SEKHAR
ss=$?

dtStop=$(date +'%Y-%m-%d %H:%M')
printf "Uptime and load avg:%s\n" "$(uptime)"
printf "%s\tStarted at %s and stopped at %s with status %d\n\n" "$dtStop" "$dtStart" "$dtStop" $ss

The reasoning here is that if it's cron terminating the task, you're unlikely to get a "finished" message, but if it's the python job then you'll get an exit status and final message reported by the wrapper. Armed with this information you can better focus your investigations.

0

I always wondered why each cron job bumps the process number up by 3. I researched the process tree to see how parenting might kill a cron task.

$ crontab -l | grep 787
11 11 17 * * sleep 787
$ ps -ef | awk 'NR == 1 || /(685|380[0-9])/'
UID        PID  PPID  C STIME TTY          TIME CMD
root       685     1  0 10:31 ?        00:00:00 /usr/sbin/cron -f
root      3808   685  0 11:11 ?        00:00:00 /usr/sbin/CRON -f
paul      3809  3808  0 11:11 ?        00:00:00 /bin/sh -c sleep 787
paul      3810  3809  0 11:11 ?        00:00:00 sleep 787
paul      3914  3720  0 11:15 pts/1    00:00:00 awk NR == 1 || /(685|380[0-9])/
$ 

10:31 is my boot time, so process 685 is my initial cron daemon.

For each job, cron launches a wrapper child CRON (pid 3808 here) that takes care of mailing any output, logging the outcome, and so on.

That executes a child shell (pid 3809) to run the crontab command itself.

Pid 3810 is the command defined by the user in crontab.

Pid 3914 is reporting this section of the process tree (reporting itself because 685 is in its args). I had to find the actual pids first (by grepping the full ps list for '787').

Any of 685, 3808 or 3809 could signal its children to take down a process, but I have never seen this done by cron (I have seen a process exceed CPU and be signalled by the shell). However, you could devise some debug using this information: for example, run free and ps for your python code, appended to a log every 10 seconds, and see whether memory or CPU becomes an issue.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .