9

I am currently using rsync to transfert large file from a host to another. I would like to limit the debit for this transfert. After reading the man page I found the option --bwlimit which is expecting a KB/s value.

I want to limit the transfert to arround 100Mbit/s So I used the following settings :

rsync --progress -v --bwlimit=10000 --temp-dir=/tmp srv:/tmp/* /tmp/

But this doesn't seems to be respected according to the output of rsync :

file.sql.gz
   570851702 100%   33.90MB/s    0:00:16 (xfer#5, to-check=34/39)
file.sql.gz
    11195140 100%    7.45MB/s    0:00:01 (xfer#6, to-check=33/39)
file.sql.gz
    51439169 100%   19.41MB/s    0:00:02 (xfer#7, to-check=32/39)
file.sql.gz
       34022 100%    1.55MB/s    0:00:00 (xfer#8, to-check=31/39)
file.sql.gz
     2305698 100%    6.96MB/s    0:00:00 (xfer#9, to-check=30/39)
file.sql.gz
     2152942 100%    2.73MB/s    0:00:00 (xfer#10, to-check=29/39)
file.sql.gz
    86655633 100%   19.33MB/s    0:00:04 (xfer#11, to-check=28/39)
file.sql.gz
  3614476032  40%   34.82MB/s    0:02:28

If I'm not wrong : 19.33MB/s ~= 154Mbit/s

So what is wrong here ?

PS : rsync version 3.0.6 protocol version 30

Thanks for your help !

  • Here it works with rsync 3.0.4 (Ubuntu), 3.0.6 (openSUSE) and 3.0.9 (Ubuntu) using the same parameters. You have to provide more information about your system. – scai Mar 12 '13 at 15:42
  • Does this help? stackoverflow.com/questions/10064316/… – mreithub Mar 12 '13 at 17:32
  • @mreithub The answer in this post is indeed interesting but actually Hugo seems to transfer rather large files. – scai Mar 12 '13 at 18:52
  • rsync bwlimits for each file, to limit bw you should limit connection , (on linux using iptables+tc atmail.com/kb/2009/throttling-bandwidth), so no easy way. – zb' Mar 13 '13 at 2:04
  • Hello, I am running RHEL 6 but this doesn't make much difference. Yes this is large file, so debit should be almost stable. For information the equivalent command with SCP is working great. So I don't know what is wrong with rsync. – Hugo Mar 13 '13 at 10:30
9

I think you experienced some "burstiness" as mentioned in the manpage:

--bwlimit=RATE
          ... 
          Some "burstiness" may be seen where rsync writes out a block 
          of data and then sleeps to bring the average rate into compliance.

So as I asked before the overall ratio at the end of your job was at last limited to 10M.

3

As mentioned in rsync man page, it transfer in KByte/sec:

--bwlimit=KBPS          limit I/O bandwidth; KBytes per second
  • 2
    That's what I used : 10 000 kilobytes = 78.12500 megabits – Hugo Dec 10 '13 at 13:33
  • 2
    You didn't pay attention to what i said. The rsync use Byte instead of bit, so you have to calculate the bandwidth in bit and covert it to byte – shgnInc Mar 8 '14 at 6:07
  • I have the same issues, when i put --bwlimit=100, i obtain 275kB/s, very regular between 265-285 kB/s, no burstiness. What is wrong in here? – Brethlosze Jul 14 '17 at 15:05
1

I want to limit the transfer to arround 100Mbit/s So I used the following settings:

rsync --progress -v --bwlimit=10000 --temp-dir=/tmp srv:/tmp/* /tmp/

100 Mbit/s is (approximately) 10 MByte/s, which is indeed 10,000 KByte/s. If you're hitting rsync burstiness then maybe the separate tool trickle will help:

trickle -s -d 10000 -u 10000 -t 10 -l 100 rsync --progress -v --temp-dir=/tmp srv:/tmp/* /tmp/
0

The value shown is the effective transfer rate which includes speeds up from compression and the delta algorithm, this can be easily observed by seeing large numbers if an interrupted rsync session is restarted with partial transfers enabled (ex --append-verify).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.