Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I am trying to download some files by using wget but after an hour my download speed decrease. And if I cancel downloading and restart it, download speed will increase for an hour.

Is it possible to put a lower bound for download speed in wget, so that when speed becomes lower than that threshold, wget restarts or gets restarted?

share|improve this question
    
Did you happen to be downloading Apple EPF feed at the time? :) –  mr.b Aug 19 '12 at 0:32

3 Answers 3

You should solve the underlying problem instead. So analyse why your connection speed drops and remove the cause.

If it is some power-saving functionality you can almost certainly switch it off or configure it in a way where your network is not affected.

share|improve this answer
    
It's not likely you can solve the problem if it's not your machine, and it could be anywhere. Throttling ISPs are a good example. –  Attila O. Mar 12 '12 at 5:22
    
@AttilaOláh why not "your machine"? I see no hint about cloud or anything else in ther question. But the same is true for "not your network" - but I have seen such effects often compined with power-save functionality (on the machine - not the network). –  Nils Mar 12 '12 at 11:23
    
Yes, good point about power-save. It's just that I ended up at this page because my ISP seems to be throttling downloads of a single HTTP request (or maybe on the TCP level, if they're counting packets in a connection). Some "advanced" download managers have the functionality to use many requests in parallel and restart a thread if speed drops below a specified threshold. I say "advanced" in quotes because I hate the GUI crap that comes with those download managers, and they're not even scriptable, so I still prefer wget. –  Attila O. Mar 12 '12 at 12:12

It's likely that either the site you're downloading from or your ISP throttles your bandwidth after a while. It may help to cap the transfer rate with --limit-rate.

There are a few options that tell wget to bail out. Pass --tries (-t) to control the number of retries. You may get better results if you pace out the retries a bit with --wait and perhaps --wait-random. Wget will automatically resume where it stopped if the server supports it. If the transfer rate is throttled to a very low figure, then --read-timeout with a very small parameter, say 0.1 for a tenth of a second, will restart the connection if the rate drops below one packet per 0.1s. Note that the connection will also be restarted if you receive no packet for that length of time due to a network glitch.

If you know in advance approximately how much time you have before getting throttled, then write a shell snippet that kills the wget process and launches it again with the -c option to resume downloading. Warning, untested; working with background subprocesses in the shell is a bit wonky, so I recommend using Perl or Python for serious work.

while wget -q -c http://example.com/wibble & wget_pid=$!
      { sleep 300; kill $wget_pid; } & kill_pid=$!
      wait $wget_pid
      kill $kill_pid
      wait
do sleep 10; done
share|improve this answer

My version of wget supports retrying after no-download timeout using read-timeout like this...

$ wget -c --read-timeout=5 http://thefile.com/getme.txt

36% [+++++++++++++++++++++++++++++++++++++++++++======>                                                                                        ] 280,061,984 --.-K/s   in 6m 16s  

2012-05-09 16:04:23 (105 KB/s) - Read error at byte 280061984/761323675 (Operation timed out). Retrying.

There seems to be no way to set the lower limit, but it heled me a lot on continuing downloads - and hopefully other people searching this issue.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.