I need to read a large log file and send it over a local network using (netbsd) netcat between two VMs on the same host workstation.

I know that netcat has an interval, but as far as I can tell, the smallest interval you can use is 1 line/second.

Most of the files I need to send this way have hundreds of thousands of lines, and some close to a million lines, so one line per second isn't feasible.

If I just use cat, my host computer/workstation winds up getting bogged down to the point of being unusable.

Using bash and common *nix tools, is there a way I can send the files, but feed it to netcat at a rate of say, 5-10 lines/second or something like that?

The end goal of this is to allow me to do some proof of concept testing for a centralized log database I am considering.

  • 2
    The simplest solution is probably to insert pv in front of netcat. Blast away at pv at full speed and it will do the rate limiting for you. Also this question may be a duplicate of this one about bandwidth limiting, or this one about rate-limiting stdout.
    – jw013
    Commented May 18, 2012 at 14:56
  • I was looking at pv, but I only see a way to limit in terms of bytes/kb/mb/etc. I need to limit on a lines per second basis, regardless of how many bytes those lines may be.
    – Lee Lowder
    Commented May 18, 2012 at 15:00
  • 2
    If, as you state in your question, you are trying to avoid network congestion, consider the following: (1) if you are using TCP, TCP already has built-in congestion control to use the maximum reasonable amount of bandwidth without clogging the network, and (2) if you need further limiting, it would make more sense to limit bandwidth using units of actual bandwidth. A "line" is a variable unit, and I'm not sure what you would accomplish by using it. If you really want to use lines as your unit for some reason, then make that explicit in the question.
    – jw013
    Commented May 18, 2012 at 15:09
  • My apologies, I have updated the question to clarify and add more detail.
    – Lee Lowder
    Commented May 18, 2012 at 15:24

4 Answers 4


pv, which is available as a NetBSD package, lets you limit the rate of a pipe.

<large_file pv -L 1k | netcat …

As it turns out, pv is the perfect tool here (at least pv 1.6.0). With the -l (line mode) flag, -L limits become lines.

Simple proofs:

seq 0 1000|pv -l -L100 > /dev/null
1k 0:00:10 [99.6 /s] [           <=>                  

seq 0 10000|pv -l -L1k > /dev/null
10k 0:00:09 [1.03k/s] [          <=>
  • and to comment on my own answer because I got bit by it by not paying attention even though I knew better: pv 1.1.4 does NOT do line limiting for -l -L ...
    – keen
    Commented Aug 17, 2017 at 20:36

If you use bash and pipes, and are looking for an easy and dirty solution, you can try using sleep.

You can use this which act like cat but with a pause at each line. while read i; do echo "$i"; sleep 0.01; done. Here is an example at a little less than 100 lines per second.

$ time (seq 1 100 | while read i; do echo "$i"; sleep 0.01; done)
real    0m1.224s
user    0m0.012s
sys     0m0.052s
  • 3
    That may alter the input. In such situations always 1) set IFS to empty string to avoid word splitting; 2) use -r to disable backslash escaping. Take this as a test string: “\tes\\t”.
    – manatwork
    Commented May 18, 2012 at 15:42
  • While this isn't the solution I had in mind, it very much did the job and is what I am using. Thank you.
    – Lee Lowder
    Commented May 18, 2012 at 16:03

Maybe cpulimit is useful. It limits the percentage of cpu time a process can use. Most distributions have it available from repositories. Problem is though that cat and netcat may use too little cpu to be really useful to throttle.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .