How to troubleshoot failing cron job

Question

I have a crontab root file that looks like this:

lab-1:/var/www/cdd# crontab -l
# do daily/weekly/monthly maintenance
# min   hour    day     month   weekday command
*       *       *       *       *       /etc/scripts/script1
*/15    *       *       *       *       /etc/scripts/script2
0       *       *       *       *       /etc/scripts/script3

I can see that all the jobs are triggered by running this command:

lab-1:/var/www/cdd# cat /var/log/messages | grep cron.info
Mar 15 13:00:00 lab-1 cron.info crond[7897]: USER root pid 26217 cmd /etc/scripts/script2
Mar 15 13:00:00 lab-1 cron.info crond[7897]: USER root pid 26219 cmd /etc/scripts/script3        
Mar 15 13:01:00 lab-1 cron.info crond[7897]: USER root pid 26293 cmd /etc/scripts/script1     

The problem is that script3 (I've proven that script2 and script1 work) is not actually producing the expected output. It's supposed to create files in another folder. However, when I run it manually like so, it works just fine:

 lab-1:/etc/scripts# bash script3

I'm not a real sys admin so not too sure what the best way is to go about troubleshooting this.

First thing that comes to mind is permissions.

lab-1:/etc/scripts# ls -lah                            
total 24                                                                     
drwxr-xr-x    2 root     root        4.0K Mar 15 12:20 .                     
drwxr-xr-x   34 root     root        4.0K Mar 14 17:11 ..                    
-rwxr-xr-x    1 root     root        5.0K Mar 15 12:19 script3
-rwxr-xr-x    1 root     root        1.8K Mar 14 15:26 script1
-rwxr-xr-x    1 root     root        1.9K Mar 14 15:26 script2

Although... having said that, if it were a permissions problem would it even show up as being triggered / started in my /var/log/messages file?

How should I proceed?

EDIT 1

lab-1:/etc/scripts# ./script3 | head -n 4                                             
Working folder set to: /tmp/tmp.kOfhip                                                                        
*****Grab SQL Data from Remote Server: spp.mydomain.net *****                                             
COPY 344                                                                                                      
Warning: Permanently added 'spp.mydomain.net,10.1.1.1' (ECDSA) to the list of known hosts.           
Evaluate /tmp/tmp.kOfhip/spp.mydomain.net.db                                                          
lab-1:/etc/scripts#                                                                     

EDIT 2

This is what my script looks like:

https://paste.pound-python.org/show/90vAlrOsAYP0CtYqNWfl/

As you can see, I'm creating a temporary folder and doing all my work in there.

EDIT 3

To prove to myself that it's not because of lines like line 9, I've commented out everything except lines 1 though 15. I added line 16 that does this:

 echo "done" >> /tmp/results.txt

ANd then I changed the schedule of the job to run every two minutes from one hour. i can see that it's run 3 times already. I guess I will continue with this approach until I find something that won't work / blows up. I don't quite understand the comment made below about using a PATH variable... but I guess I will google it.

EDIT 4

I changed the crontabs root file so it outputs the results of script3 to a file and this is what i see:

Working folder set to: /tmp/tmp.GeNGDJ
*****Grab SQL Data from Remote Server: servername *****
COPY 344
Warning: Permanently added 'spp.mydomain.net,10.1.1.132' (ECDSA) to the list of known hosts.
Permission denied (publickey,keyboard-interactive).
Evaluate /tmp/tmp.GeNGDJ/spp.mydomain.net.db
cat: can't open '/tmp/tmp.GeNGDJ/spp.mydomain.net.db': No such file or directory

So it's dying while trying to scp the file. The remote SQL runs just fine and shows output. But as you can see I'm getting a permission denied But if i run the same command manually, it seems to work. Will have to keep poking around. Will try dumping the ENV like is suggested in the answer below.

Answer 1

A common mistake when writing scripts that will be executed later by cron, is that you assume the script will have exactly the same environment that you have when you are logged in and you are developing it. It hasn't!

Write a script4 which contains the following line

OFILE=/tmp/crons.environment
(/usr/bin/whoami
/usr/bin/env ) > $OFILE 2>&1

And get cron to run that

Now compare the output in /tmp/crons.environment to what you get when you just type env

Your script assumes for example that $PATH is setup correctly to find all the programs you execute, you are also querying a database, there could be further Environment variables required for those commands to run correctly.

To check the output from the cron job. Temporarily modify the command run by cron and redirect stdout and stderr to a known file, Like I did above.

0       *       *       *       *       /etc/scripts/script3 > /tmp/s3.out 2>&1