Is it possible to throttle (limit) the download speed of
Is it possible to change the throttle value while it is downloading ?
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It only takes a minute to sign up.Sign up to join this community
Yes both wget and curl support limiting your download rate. Both options are directly mentioned in the man page.
--limit-rate <speed> Specify the maximum transfer rate you want curl to use. This feature is useful if you have a limited pipe and you'd like your transfer not to use your entire bandwidth. The given speed is measured in bytes/second, unless a suffix is appended. Appending 'k' or 'K' will count the number as kilobytes, 'm' or M' makes it megabytes, while 'g' or 'G' makes it gigabytes. Examples: 200K, 3m and 1G.
curl --limit-rate 423K
--limit-rate=amount Limit the download speed to amount bytes per second. Amount may be expressed in bytes, kilobytes with the k suffix, or megabytes with the m suffix. For example, --limit-rate=20k will limit the retrieval rate to 20KB/s. This is useful when, for whatever reason, you don't want Wget to consume the entire available bandwidth.
2 years later I will throw this tidbit in, while
curl are not interactive, at least
wget (and possibly
curl but i do not know for sure) has the
-c switch (which stands for continue from where I left off downloading earlier). So if you need to change your speed in the middle of a download and you presumably used the
-c switch with the
--limit-rate=x then you could stop
wget and restart it with a different speed and it would change.
It is possible to limit the traffic rate using the
netem tools but this will limit the rate for the network interface of the computer. I am assuming that you use only
curl and no other application is exchanging traffic through the network interface.
tc uses Token Bucket Filter (TBF) to control the rate.
One example of TBF would be as follows (ref. http://www.lartc.org/manpages/tc-tbf.html):
To attach a TBF with a sustained maximum rate of 0.5mbit/s, a peakrate of 1.0mbit/s, a 5kilobyte buffer, with a pre-bucket queue size limit calculated so the TBF causes at most 70ms of latency, with perfect peakrate behaviour, issue:
# tc qdisc add dev eth0 root tbf rate 0.5mbit \ burst 5kb latency 70ms peakrate 1mbit \ minburst 1540
Another example of usign tc and netem would be as follows (found in http://www.linuxfoundation.org/collaborate/workgroups/networking/netem):
There is no rate control built-in to the netem discipline, instead use one of the other disciplines that does do rate control. In this example, we use Token Bucket Filter (TBF) to limit output.
To add the delay of each packet going/coming through the interface eth0
# tc qdisc add dev eth0 root handle 1:0 netem delay 100ms
to add the data rate in tbf, packet buffer size and maximum burst limit
# tc qdisc add dev eth0 parent 1:1 handle 10: tbf rate 256kbit buffer 1600 limit 3000
To see the list of rules assigned in tc for interface eth0
# tc -s qdisc ls dev eth0
The output of the above command would be as below
qdisc netem 1: limit 1000 delay 100.0ms Sent 0 bytes 0 pkts (dropped 0, overlimits 0 ) qdisc tbf 10: rate 256Kbit burst 1599b lat 26.6ms Sent 0 bytes 0 pkts (dropped 0, overlimits 0 )
Check on the options for buffer and limit as you might find you need bigger defaults than these (they are in bytes)
For wget, when downloading multiple pages at once, I find the
--wait option more understandable. It makes wget wait a certain number of seconds between requests, which is more consistent with
Crawl-delay directive of robots.txt.
-w seconds --wait=seconds Wait the specified number of seconds between the retrievals. Use of this option is recommended, as it lightens the server load by making the requests less frequent. Instead of in seconds, the time can be specified in minutes using the "m" suffix, in hours using "h" suffix, or in days using "d" suffix. Specifying a large value for this option is useful if the network or the destination host is down, so that Wget can wait long enough to reasonably expect the network error to be fixed before the retry. The waiting interval specified by this function is influenced by "--random-wait", which see.