0

I have the following crontab to pull an online resource at 9am every day and write it to a file locally:

0 9 * * * /usr/bin/wget --http-user=username --http-password=password https://remote_file_to_pull.json -O /local_output_file.json >/dev/null 2>&1

The problem is that sometimes when retrieving the remote file, the server throws a Error 500. What wget does is write over an empty file.

Is there a way to change the cron to only write the file if wget does not encounter an error and/or the file to write is not empty?

Thanks!

1
  • 2
    I don't know that wget can do what you ask, but if you were to use a temp file and replace the real file on success you wouldn't need wget to do anything different. Jan 4 at 13:22
0

Replace your crontab entry with a reference to this script, which updates the output only on success from wget and where it returns a file with non-zero length

#!/bin/bash
#
url='https://remote_file_to_pull.json'
user='username'
pass='password'

target='/local_output_file.json'

if wget --quiet --http-user="$user"--http-password="$pass" -O "$target.tmp" "$url" && [[ -s "$target.tmp" ]]
then
    # Success
    mv -f "$target.tmp" "$target"
else
    # Failure of some sort; you might want to report this
    :
fi
rm -f "$target.tmp"

By the way. I'm hoping your code was illustrative, but it's generally not good practice to write files directly to the root of the filesystem.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.