0

I have a bash script which loops through a list of filenames in a text file, deduces a date from the filename and then calls a library script (./dropbox_uploader.sh delete [filname]) to conditionally delete a remote file in my Dropbox account using Dropbox's api. The condition is simply whether the file is older than [n] days. n is passed into the script in position $2. $1 is a text file containing the entries.

2 local variables, $go and $stay count the instances of where a file is eligible for deletion, or not. At the end of my script, I fire off an email via the mail program using the values of the 2 variables..

My problem is that if I run this from the command line everything works just fine and the email reads:

"The dropbox files were purged: 554 files deleted and 310 were kept."

However, if the script is run via Cron using my user's crontab then it seems that the local variables are both zero / null, and the email is:

The dropbox files were purged: 0 files deleted and 0 were kept.

I can run the script again manually after the cron run and the variables are non zero.

I think I'm missing something fundamental about the way cron runs for the user and would be grateful for suggestions.

My script:

#!/bin/bash
threshold=$(date -d "$2 days ago" +%s)
go=0
stay=0
rm /home/pi/Dropbox-Uploader/camDeleteLog.txt
echo "deleting older than $2 days ago ..."

while IFS='' read -r line || [[-n "$line" ]]; do
        line=$(echo $line | xargs)      # trim spaces from the filename, just i$
        y=${line:0:4}
        m=${line:5:2}
        d=${line:8:2}
        #echo "Filedate is $y-$m-$d"
        seconds=$(date -d "$y-$m-$d" +%s)

        if ((seconds < threshold))
        then
                echo "Deleting file $go, $line"
                ((go++))
                ###echo "./dropbox_uploader.sh delete \"$line\""
               ./dropbox_uploader.sh delete "$line" >> /home/pi/Dropbox-Upload$

        else
                echo "$line is too new to delete"
                ((stay++))
        fi
done < "$1"
echo "The dropbox files were purged: $go files deleted and $stay were kept." | $

My crontab for user pi:

0 */8 * * * /home/pi/Dropbox-Uploader/moveVideostoDropbox.py # JOB_ID_1
0 0,6,12,18 * * * Dropbox-Uploader/processDBFiles.sh # JOB_ID_2
0 1,7,13,19 * * * Dropbox-Uploader/processByDate.sh camfiles.txt 2 # JOB_ID_3

Job 3 is the one that does the date processing and deleting. Jobs 2 and 3 were originally run together but I suspected (wrongly) that this was the root of my problem so split the 2 and ran them an hour apart as an experiement - Job 2 will never take more than 1 hour to run.

  • You are using relative paths in the script. Do you know what directory is your working directory when the script is run by cron? You could have cd "$HOME/somedir" early in your script to make sure that you are in the correct directory when the script runs. – Kusalananda Jan 28 at 15:24
  • OK, helpful replies, thank you. I will go and do some more investigating. Incidentally, although I have resorted back to editing cron using "crontab -e", there is a cron management app I installed on my Pi. When you edit a script via the app it warns: "Note about working directory of executed tasks: Recurrent tasks will be run from the home directory." This is the reason I referenced the script to run from the home/pi directory. But I will look at making sure I've got proper absolute paths. – MMS Jan 28 at 15:53
  • I wonder if the pi uses sh by default for cron jobs? – Jeff Schaller Jan 28 at 15:54
  • Jeff - good point. Researching how to determine cron shell has taken me to raspberrypi.stackexchange.com/a/2167. I'll follow that up too. Thanks. – MMS Jan 28 at 16:00
0

The problem is that the while-loop is implemented in a subshell. Here's a thead that discusses it and a potential proper solution.

But I've been frustrated by this problem enough that I try to avoid it altogether. Using a file to hold variables won't win points for elegance, but it works. This little script illustrates it.

#!/bin/bash
c=0
FILE=/tmp/$(basename $0).$$
while IFS= read -r line; do
    ((c++))
    echo "count=$c" > $FILE
done < somefile
. $FILE
echo "Read $count lines"
rm -f $FILE
  • OK, as Ken Jackson says, the while loop is implemented in a subshell. I've now written my variables to a file inside the loop and then later used the result from calls to AWK to echo the values into the text string which is the body of the email. What I can't now figure out is why the while loop variables are valid outside the loop when I run the script manually from the shell, but not when triggered by Cron. I think that's a separate question! – MMS Jan 29 at 14:19
0

I have experienced similar issues with cron jobs. I cannot specifically say what is wrong with your cron entry, but I will share some thoughts of what I have noticed about the differences between cron running a script, and an interactive shell running a script.

First, cron does NOT read your local environment, the default PATHis, I believe, /usr/bin:/usr/sbin. One test to try is to print your environment variables at the beginning of the script and run interactively, then run from the cron and see the difference.

#!/bin/bash
env

Second, do not assume that you are running from your home directory. Specify ALL path names. i.e. /bin/date, /home/pi/scriptname.sh /home/pi/filename, etc.

Third, try sourcing your local environment in the script and/or try add the -f option to your shebang

#!/bin/bash
. /home/pi/.bashrc  #Not the period space at the beginning of the line

or

#!/bin/bash -f

Fourth, check the log output of the script to be sure it is processing the right files and actually setting your variables. You can do this by reading the cron log (wherever you have that set to log to), or redirect STDIN & STDOUT to a file:

0 1,7,13,19 * * * /home/pi/Dropbox-Uploader/processByDate.sh /home/pi/camfiles.txt 2 >/dev/null 2>&1
  • 1
    The PATH may be different, but it will be sane, i.e. there's definitely no need to specify full paths to standard utilities like date. Also note that if the script needs to have access to other directories usually found in PATH, it is free to modify that variable itself. Sourcing .bashrc from a script is not really a good solution as that file is supposed to set up an interactive environment, which may include starting things like screen or tmux... – Kusalananda Jan 28 at 15:36
  • Permanently sourcing .bashrc is, as you say, not a good idea. This is only meant to be a troubleshooting step if it works when sourcing your local environment then your environment contains something your script needs. There is a .bashrc and a .bash_profile. One is for interactive, the other is for non-interactive. I'd assume the OP knows if his profile I have seen instances where they did not work correctly without them. It doesn't hurt to do it. Some standard utilities are both builtin and executable file. Specifying a path forces to executable (consider the echo command). – Scottie H Jan 28 at 15:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.