Below is part of a script I use to download files from a website daily. However recently they added speed limits to downloading files. I increased the sleep time but then everything else takes too long, there are a lot of files to download and some are very small.

I want to remove the sleep wait time or set it very low and modify the script so that it will wait for the file to finish downloading


I have found the cause of the larger files not completing their download. Failure when receiving data from the peer how can I fix this? I have read switching to wget is the best option but how would this script work with wget?

#check directories are empty, not empty if there was a problem last time
cd /home/user/upload
if [ "$(ls -A /home/user/upload)" ]; then
#     echo 'Directory not empty error for csv manipulation' | /bin/mailx -s "Server scrapeandcleandomains error" use
     echo "$(date) Directory /home/user/upload not empty for csv manipulation"  >> /home/user/logfile
     exit 1
     echo $(date) starting normal >> /home/user/logfile

#create yesterday variable
yesterday=$(echo  $(date --date="$1 - 2 days" +"%Y_%m_%d" ) )
#$(date --date="-2 day" +"%Y_%m_%d")

#download .csv.gz files (old wget command) OBSOLETE!!!!!
#cd /home/user/upload
#wget -R html,"index.*" -A "$yesterday*.csv.gz" -N -r -c -l1 -nd --no-check-certificate --user USERNAME --password PASSWORD -np http://www.websitedownloadfrom.com/sub/
#exit 1

#download index and sanitize > index2.tmp
cd /home/user
curl -u "USERNAME:PASSWORD" -k  http://www.websitedownloadfrom.com/sub/ -o index.html.tmp
links -dump index.html.tmp > /home/user/index.tmp
#this will work until 2049 ONLY!!
sed -i '/20[1-4][0-9]/!d' index.tmp
sed -i '/\[DIR\]/d' index.tmp
for i in {1..50} ; do
   sed -i 's/  / /' index.tmp

awk -F" " '{ print $3 }' index.tmp > index2.tmp
sed -i "/^${yesterday}/!d" index2.tmp

#download .csv.gz files according to index2.tmp
while read F  ; do
    cd /home/user/upload
    curl -u "USERNAME:PASSWORD" -k  http://www.websitedownloadfrom.com/sub/$F -o $F &
    sleep 80
done < /home/user/index2.tmp

sleep 60

#check that we downloaded something
cd /home/user/upload
if ! [ "$(ls -A /home/user/upload)" ]; then
    echo 'nothing downloaded from upload'  >> /home/user/logfile
    rm -f /home/user/upload/*
    rm -f /home/user/index.html.tmp
    rm -f /home/user/index.tmp
    rm -f /home/user/index2.tmp
    exit 1

Remove the sleep 80 command and the & from the curl command immediately prior to it. Removing the & will make the script wait for the curl download to finish before proceeding to the next pass through the loop.

  • Thank you that works but I have just found the cause of the download to stop working on the larger files. I am getting "Failure when receiving data from the peer" error. I have read this is a common problem with curl and people say to switch to wget? How can I switch to wget or modify the curl command to fix this – Teddy291 Jul 6 '15 at 17:18
  • 1
    See man wget. – John Jul 6 '15 at 17:21
  • Is it possible to set curl to retry the connection multiple times if it fails and resume? – Teddy291 Jul 6 '15 at 17:26
  • 1
    Yes - see man curl. – John Jul 6 '15 at 17:27
  • curl -u "USERNAME:PASSWORD" -k -connect-timeout 15 websitedownloadfrom.com/sub/$F -o $F - Will that work? Timeout to 15 seconds – Teddy291 Jul 6 '15 at 17:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.