12

tr seems to buffer its input so that this command LongRunningCommand|tr \\n , will only start producing output after a few kilobytes of input from LongRunningCommand have accumulated.

Is there a way to force tr to stop this buffering or any other command that can replace new-lines with an other character without buffering?

P.S. I've already tried the first two suggestions from Turn off buffering in pipe without success.

Improve this question
8
  • 1
    did you apply stdbuf to LongRunningCommand or to tr, or to both, differently?
    – meuh
    Oct 27, 2015 at 15:09
  • To both like this: stdbuf -o0 fping -aAq -r2 -g 10.30.0.1 10.30.0.255 2>/dev/null | stdbuf -i0 tr \\n ,
    – ndemou
    Oct 27, 2015 at 15:15
  • fping -q says "Don't show per-probe results, but only the final summary", so perhaps it only does one long write at the end?
    – meuh
    Oct 27, 2015 at 15:20
  • No the fping command works fine by itself. Many thanks for the suggestions though
    – ndemou
    Oct 27, 2015 at 15:28
  • 2
    I think the buffering problem is actually in the output of tr. Try |stdbuf -i0 -o0 tr ...
    – meuh
    Oct 27, 2015 at 15:36

2 Answers 2

Reset to default
15

Commands generally don't buffer their input. They would do a read() for a large chunk, but when reading from a pipe, if there aren't that many bytes in the pipe, the read() system call will return with as many characters there are and the application will generally work with that if it can.

A notable exception to that is mawk which will keep re-read()ing until the input buffer is full.

Applications do buffer their output (stdout) though. The usual behaviour is that if the output is going to a tty, then the buffering will be line-wise (that is, it won't start writing to stdout until it has a full line to output, or a block-full for very long line), while for every other type of file, the buffering is by blocks (that is, it won't start writing until is has one block full to write (something like 4KiB/8KiB... depends on the software and system)).

So in your case LongRunningCommand likely buffers its output by blocks (since its output is a pipe and not a tty), and tr likely buffers its output by line since its output is probably the terminal.

But, since you remove every newline character from its output, it will never output a line, so the buffering will be by block.

So here you want to disable buffering for both LongRunningCommand and tr. On GNU or FreeBSD systems:

stdbuf -o0 LongRunningCommand | stdbuf -o0 tr '\n' ,

Note that if you want to join the lines with a comma, a better approach is to use paste -sd , -. That way the output will be terminated by a newline character (you'll probably still need to disable buffering).

Improve this answer
3
  • @mikeserv. Yes, and one that uses stdio buffering and doesn't call setvbuf/setbuffer... by itself and is not a shell builtin. But generally on GNU and FreeBSD systems, all those conditions are satisfied.
    – Stéphane Chazelas
    Oct 28, 2015 at 15:48
  • oh. i only learned that myself a few days ago when i was disappointed that i couldnt affect the i/o streams for my statically compiled sed. sorry if it was annoying - but i didnt realize it was general knowledge. i guess it uses LD_PRELOAD.
    – mikeserv
    Oct 28, 2015 at 15:54
  • Thanks for the thorough explanation of what happens Stéphane. Greatly appreciated
    – ndemou
    Oct 28, 2015 at 16:26
6

To replace newlines with ",", you can run

awk '{ printf "%s,", $0 }'

GNU awk (gawk) and Solaris nawk will run with line buffering on stdin and no buffering stdout when the output is to a terminal. If your awk is mawk, which happens on Ubuntu, you can give it the -W interactive option to get that same buffering behavior.

Improve this answer
0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .