lower bound for download speed in wget

Question

I am trying to download some files by using wget but after an hour my download speed decrease. And if I cancel downloading and restart it, download speed will increase for an hour.

Is it possible to put a lower bound for download speed in wget, so that when speed becomes lower than that threshold, wget restarts or gets restarted?

Answer 1

You should solve the underlying problem instead. So analyse why your connection speed drops and remove the cause.

If it is some power-saving functionality you can almost certainly switch it off or configure it in a way where your network is not affected.

Answer 2

It's likely that either the site you're downloading from or your ISP throttles your bandwidth after a while. It may help to cap the transfer rate with --limit-rate.

There are a few options that tell wget to bail out. Pass --tries (-t) to control the number of retries. You may get better results if you pace out the retries a bit with --wait and perhaps --wait-random. Wget will automatically resume where it stopped if the server supports it. If the transfer rate is throttled to a very low figure, then --read-timeout with a very small parameter, say 0.1 for a tenth of a second, will restart the connection if the rate drops below one packet per 0.1s. Note that the connection will also be restarted if you receive no packet for that length of time due to a network glitch.

If you know in advance approximately how much time you have before getting throttled, then write a shell snippet that kills the wget process and launches it again with the -c option to resume downloading. Warning, untested; working with background subprocesses in the shell is a bit wonky, so I recommend using Perl or Python for serious work.

while wget -q -c http://example.com/wibble & wget_pid=$!
      { sleep 300; kill $wget_pid; } & kill_pid=$!
      wait $wget_pid
      kill $kill_pid
      wait
do sleep 10; done

Answer 3

My version of wget supports retrying after no-download timeout using read-timeout like this...

$ wget -c --read-timeout=5 http://thefile.com/getme.txt

36% [+++++++++++++++++++++++++++++++++++++++++++======>                                                                                        ] 280,061,984 --.-K/s   in 6m 16s  

2012-05-09 16:04:23 (105 KB/s) - Read error at byte 280061984/761323675 (Operation timed out). Retrying.

There seems to be no way to set the lower limit, but it heled me a lot on continuing downloads - and hopefully other people searching this issue.