Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I need to read a large log file and send it over a local network using (netbsd) netcat between two VMs on the same host workstation.

I know that netcat has an interval, but as far as I can tell, the smallest interval you can use is 1 line/second.

Most of the files I need to send this way have hundreds of thousands of lines, and some close to a million lines, so one line per second isn't feasible.

If I just use cat, my host computer/workstation winds up getting bogged down to the point of being unusable.

Using bash and common *nix tools, is there a way I can send the files, but feed it to netcat at a rate of say, 5-10 lines/second or something like that?

The end goal of this is to allow me to do some proof of concept testing for a centralized log database I am considering.

share|improve this question
2  
The simplest solution is probably to insert pv in front of netcat. Blast away at pv at full speed and it will do the rate limiting for you. Also this question may be a duplicate of this one about bandwidth limiting, or this one about rate-limiting stdout. –  jw013 May 18 '12 at 14:56
    
I was looking at pv, but I only see a way to limit in terms of bytes/kb/mb/etc. I need to limit on a lines per second basis, regardless of how many bytes those lines may be. –  Lee Lowder May 18 '12 at 15:00
2  
If, as you state in your question, you are trying to avoid network congestion, consider the following: (1) if you are using TCP, TCP already has built-in congestion control to use the maximum reasonable amount of bandwidth without clogging the network, and (2) if you need further limiting, it would make more sense to limit bandwidth using units of actual bandwidth. A "line" is a variable unit, and I'm not sure what you would accomplish by using it. If you really want to use lines as your unit for some reason, then make that explicit in the question. –  jw013 May 18 '12 at 15:09
    
My apologies, I have updated the question to clarify and add more detail. –  Lee Lowder May 18 '12 at 15:24
add comment

3 Answers

up vote 1 down vote accepted

If you use bash and pipes, and are looking for an easy and dirty solution, you can try using sleep.

You can use this which act like cat but with a pause at each line. while read i; do echo "$i"; sleep 0.01; done. Here is an example at a little less than 100 lines per second.

$ time (seq 1 100 | while read i; do echo "$i"; sleep 0.01; done)
[...]
real    0m1.224s
user    0m0.012s
sys     0m0.052s
share|improve this answer
2  
That may alter the input. In such situations always 1) set IFS to empty string to avoid word splitting; 2) use -r to disable backslash escaping. Take this as a test string: “\tes\\t”. –  manatwork May 18 '12 at 15:42
    
While this isn't the solution I had in mind, it very much did the job and is what I am using. Thank you. –  Lee Lowder May 18 '12 at 16:03
add comment

pv, which is available as a NetBSD package, lets you limit the rate of a pipe.

<large_file pv -L 1k | netcat …
share|improve this answer
add comment

Maybe cpulimit is useful. It limits the percentage of cpu time a process can use. Most distributions have it available from repositories. Problem is though that cat and netcat may use too little cpu to be really useful to throttle.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.