Below is part of a script I use to download files from a website daily. However recently they added speed limits to downloading files. I increased the sleep time but then everything else takes too long, there are a lot of files to download and some are very small.

I want to remove the sleep wait time or set it very low and modify the script so that it will wait for the file to finish downloading


I have found the cause of the larger files not completing their download. Failure when receiving data from the peer how can I fix this? I have read switching to wget is the best option but how would this script work with wget?

#check directories are empty, not empty if there was a problem last time
cd /home/user/upload
if [ "$(ls -A /home/user/upload)" ]; then
#     echo 'Directory not empty error for csv manipulation' | /bin/mailx -s "Server scrapeandcleandomains error" use
     echo "$(date) Directory /home/user/upload not empty for csv manipulation"  >> /home/user/logfile
     exit 1
     echo $(date) starting normal >> /home/user/logfile

#create yesterday variable
yesterday=$(echo  $(date --date="$1 - 2 days" +"%Y_%m_%d" ) )
#$(date --date="-2 day" +"%Y_%m_%d")

#download .csv.gz files (old wget command) OBSOLETE!!!!!
#cd /home/user/upload
#wget -R html,"index.*" -A "$yesterday*.csv.gz" -N -r -c -l1 -nd --no-check-certificate --user USERNAME --password PASSWORD -np http://www.websitedownloadfrom.com/sub/
#exit 1

#download index and sanitize > index2.tmp
cd /home/user
curl -u "USERNAME:PASSWORD" -k  http://www.websitedownloadfrom.com/sub/ -o index.html.tmp
links -dump index.html.tmp > /home/user/index.tmp
#this will work until 2049 ONLY!!
sed -i '/20[1-4][0-9]/!d' index.tmp
sed -i '/\[DIR\]/d' index.tmp
for i in {1..50} ; do
   sed -i 's/  / /' index.tmp

awk -F" " '{ print $3 }' index.tmp > index2.tmp
sed -i "/^${yesterday}/!d" index2.tmp

#download .csv.gz files according to index2.tmp
while read F  ; do
    cd /home/user/upload
    curl -u "USERNAME:PASSWORD" -k  http://www.websitedownloadfrom.com/sub/$F -o $F &
    sleep 80
done < /home/user/index2.tmp

sleep 60

#check that we downloaded something
cd /home/user/upload
if ! [ "$(ls -A /home/user/upload)" ]; then
    echo 'nothing downloaded from upload'  >> /home/user/logfile
    rm -f /home/user/upload/*
    rm -f /home/user/index.html.tmp
    rm -f /home/user/index.tmp
    rm -f /home/user/index2.tmp
    exit 1

1 Answer 1


Remove the sleep 80 command and the & from the curl command immediately prior to it. Removing the & will make the script wait for the curl download to finish before proceeding to the next pass through the loop.

  • Thank you that works but I have just found the cause of the download to stop working on the larger files. I am getting "Failure when receiving data from the peer" error. I have read this is a common problem with curl and people say to switch to wget? How can I switch to wget or modify the curl command to fix this
    – Teddy77
    Commented Jul 6, 2015 at 17:18
  • 1
    See man wget.
    – John
    Commented Jul 6, 2015 at 17:21
  • Is it possible to set curl to retry the connection multiple times if it fails and resume?
    – Teddy77
    Commented Jul 6, 2015 at 17:26
  • 1
    Yes - see man curl.
    – John
    Commented Jul 6, 2015 at 17:27
  • curl -u "USERNAME:PASSWORD" -k -connect-timeout 15 websitedownloadfrom.com/sub/$F -o $F - Will that work? Timeout to 15 seconds
    – Teddy77
    Commented Jul 6, 2015 at 17:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .