25

I'm running CentOS 5.7 and I have a backup utility that has the option of dumping its backup file to stdout. The backup file is rather large (multiple gigabytes). The target is an SSHFS filesystem. To ensure that I don't hog the bandwidth and degrade the performance of the network, I would like to limit the speed with which data is written to the "disk".

How can I limit the ability of stdout based on a byte number? For example, limiting a process's ability to write to about 768Bps.

Improve this question
2
  • 6
    @msw this question is about rate-limiting stdout, the previous question was about limiting network traffic. Different beasts...
    – voretaq7
    Mar 14, 2012 at 19:14
  • 7
    @msw Last I checked, the point of the SE network is to collect a massive amount of knowledge in specific areas. In this case, it's about *nix. Limiting the output of stdout is different than limiting bandwidth. Just because the underlying problem is the same doesn't mean that future visitors won't find each question useful independent of the other. Remember that the Qs&As are just as much for googlers as for the actual askers, which is why deleting answered questions is such a cardinal sin here.
    – MDMarra
    Mar 14, 2012 at 19:27

3 Answers 3

Reset to default
28

You can add a rate limiting tool to your pipeline.

For example there is pv which has a rate-limiting option:

-L RATE, --rate-limit RATE

Limit the transfer to a maximum of RATE bytes per second. A suffix of "k", "m", "g", or "t" can be added to denote kilobytes (*1024), megabytes, and so on

An alternative is the tool buffer which has:

   -u microseconds

After every write pause for this many microseconds. Defaults to zero. (Surprisingly a small sleep, 100 usecs, after each write can greatly enhance throughput on some drives.)

Improve this answer
3

I would be tempted to dump this on a local disk instead of writing it directly to the remote volume. This way you can use rsync with --bwlimit to easily limit the transfer.

One of the benefits to this approach is that if there is a connectivity hiccup to the remote machine, the transfer is interrupted, but the backup itself doesn't explode in a blaze of glory.

Improve this answer
1
  • Noted. I do so wish that rsync was a viable solution in this rather limited situation I find myself in. =|
    – Wesley
    Mar 14, 2012 at 19:33
3

pv (pipe view)

echo asdlfkjasdf | pv -q -L 12
Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .