11

tr seems to buffer its input so that this command LongRunningCommand|tr \\n , will only start producing output after a few kilobytes of input from LongRunningCommand have accumulated.

Is there a way to force tr to stop this buffering or any other command that can replace new-lines with an other character without buffering?


P.S. I've already tried the first two suggestions from Turn off buffering in pipe without success.

8
  • 1
    did you apply stdbuf to LongRunningCommand or to tr, or to both, differently? – meuh Oct 27 '15 at 15:09
  • To both like this: stdbuf -o0 fping -aAq -r2 -g 10.30.0.1 10.30.0.255 2>/dev/null | stdbuf -i0 tr \\n , – ndemou Oct 27 '15 at 15:15
  • fping -q says "Don't show per-probe results, but only the final summary", so perhaps it only does one long write at the end? – meuh Oct 27 '15 at 15:20
  • No the fping command works fine by itself. Many thanks for the suggestions though – ndemou Oct 27 '15 at 15:28
  • 2
    I think the buffering problem is actually in the output of tr. Try |stdbuf -i0 -o0 tr ... – meuh Oct 27 '15 at 15:36
13

Commands generally don't buffer their input. They would do a read() for a large chunk, but when reading from a pipe, if there aren't that many bytes in the pipe, the read() system call will return with as many characters there are and the application will generally work with that if it can.

A notable exception to that is mawk which will keep re-read()ing until the input buffer is full.

Applications do buffer their output (stdout) though. The usual behaviour is that if the output is going to a tty, then the buffering will be line-wise (that is, it won't start writing to stdout until it has a full line to output, or a block-full for very long line), while for every other type of file, the buffering is by blocks (that is, it won't start writing until is has one block full to write (something like 4KiB/8KiB... depends on the software and system)).

So in your case LongRunningCommand likely buffers its output by blocks (since its output is a pipe and not a tty), and tr likely buffers its output by line since its output is probably the terminal.

But, since you remove every newline character from its output, it will never output a line, so the buffering will be by block.

So here you want to disable buffering for both LongRunningCommand and tr. On GNU or FreeBSD systems:

stdbuf -o0 LongRunningCommand | stdbuf -o0 tr '\n' ,

Note that if you want to join the lines with a comma, a better approach is to use paste -sd , -. That way the output will be terminated by a newline character (you'll probably still need to disable buffering).

3
  • @mikeserv. Yes, and one that uses stdio buffering and doesn't call setvbuf/setbuffer... by itself and is not a shell builtin. But generally on GNU and FreeBSD systems, all those conditions are satisfied. – Stéphane Chazelas Oct 28 '15 at 15:48
  • oh. i only learned that myself a few days ago when i was disappointed that i couldnt affect the i/o streams for my statically compiled sed. sorry if it was annoying - but i didnt realize it was general knowledge. i guess it uses LD_PRELOAD. – mikeserv Oct 28 '15 at 15:54
  • Thanks for the thorough explanation of what happens Stéphane. Greatly appreciated – ndemou Oct 28 '15 at 16:26
5

To replace newlines with ",", you can run

awk '{ printf "%s,", $0 }'

GNU awk (gawk) and Solaris nawk will run with line buffering on stdin and no buffering stdout when the output is to a terminal. If your awk is mawk, which happens on Ubuntu, you can give it the -W interactive option to get that same buffering behavior.

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.