Throttle the download speed of wget or curl while downloading

Question

Is it possible to throttle (limit) the download speed of wget or curl ?

Is it possible to change the throttle value while it is downloading ?

Answer 1

Yes both wget and curl support limiting your download rate. Both options are directly mentioned in the man page.

curl

   --limit-rate <speed>
          Specify the maximum transfer rate you want curl to use. 
           This feature is useful  if you  have a limited pipe and 
           you'd like your transfer not to use your entire bandwidth.

          The given speed is measured in bytes/second, unless a suffix 
          is appended.  Appending  'k'  or 'K' will count the number
          as kilobytes, 'm' or M' makes it megabytes, while 'g' or 'G' 
          makes it gigabytes. Examples: 200K, 3m and 1G.

E.g: curl --limit-rate 423K

wget

   --limit-rate=amount
       Limit the download speed to amount bytes per second.  Amount may
       be expressed in bytes, kilobytes with the k suffix, or 
       megabytes with the m suffix.  For example, --limit-rate=20k will limit 
       the retrieval rate to 20KB/s.  This is useful when, for
       whatever reason, you don't want Wget to consume 
       the entire available bandwidth.

E.g: wget --limit-rate=423k

Answer 2

2 years later I will throw this tidbit in, while wget and curl are not interactive, at least wget (and possibly curl but i do not know for sure) has the -c switch (which stands for continue from where I left off downloading earlier). So if you need to change your speed in the middle of a download and you presumably used the -c switch with the --limit-rate=x then you could stop wget and restart it with a different speed and it would change.

Answer 3

It is possible to limit the traffic rate using the tc and netem tools but this will limit the rate for the network interface of the computer. I am assuming that you use only wget or curl and no other application is exchanging traffic through the network interface.

tc uses Token Bucket Filter (TBF) to control the rate.

One example of TBF would be as follows (ref. http://www.lartc.org/manpages/tc-tbf.html):

To attach a TBF with a sustained maximum rate of 0.5mbit/s, a peakrate of 1.0mbit/s, a 5kilobyte buffer, with a pre-bucket queue size limit calculated so the TBF causes at most 70ms of latency, with perfect peakrate behaviour, issue:

# tc qdisc add dev eth0 root tbf rate 0.5mbit \ burst 5kb latency 70ms peakrate 1mbit \ minburst 1540

Another example of usign tc and netem would be as follows (found in http://www.linuxfoundation.org/collaborate/workgroups/networking/netem):

There is no rate control built-in to the netem discipline, instead use one of the other disciplines that does do rate control. In this example, we use Token Bucket Filter (TBF) to limit output.

To add the delay of each packet going/coming through the interface eth0

 # tc qdisc add dev eth0 root handle 1:0 netem delay 100ms

to add the data rate in tbf, packet buffer size and maximum burst limit

 # tc qdisc add dev eth0 parent 1:1 handle 10: tbf rate 256kbit buffer 1600 limit 3000

To see the list of rules assigned in tc for interface eth0

 # tc -s qdisc ls dev eth0

The output of the above command would be as below

 qdisc netem 1: limit 1000 delay 100.0ms
  Sent 0 bytes 0 pkts (dropped 0, overlimits 0 )
 qdisc tbf 10: rate 256Kbit burst 1599b lat 26.6ms
  Sent 0 bytes 0 pkts (dropped 0, overlimits 0 )

Check on the options for buffer and limit as you might find you need bigger defaults than these (they are in bytes)

Answer 4

For wget, when downloading multiple pages at once, I find the --wait option more understandable. It makes wget wait a certain number of seconds between requests, which is more consistent with Crawl-delay directive of robots.txt.

wget

 -w seconds
 --wait=seconds
     Wait the specified number of seconds between the retrievals.  Use of this
     option is recommended, as it lightens the server load by making the
     requests less frequent.  Instead of in seconds, the time can be specified
     in minutes using the "m" suffix, in hours using "h" suffix, or in days
     using "d" suffix.

     Specifying a large value for this option is useful if the network or the
     destination host is down, so that Wget can wait long enough to reasonably
     expect the network error to be fixed before the retry.  The waiting
     interval specified by this function is influenced by "--random-wait", which
     see.