4

I usually use command1 | command2 | command3 a lot in Linux but most of them are dealing with definite content.

When I tried this with an infinite stream cat | sed '' | sed '' which hopefully simulates an infinite stream it didn't work utill I terminated it with Ctrl-D. I can solve the problem with using cat | sed -e '' -e '' but I would like to know why the first one doesn't work. cat | cat | cat works just fine. Is it something to do with sed, if so what is that problem?

I tried to think about this problem and the only thing I found different was that when I am using cat I hit the Enter key which does something special that is not happening in the first sed '' above?

Can anyone let me know how to make pipe work seamlessly with infinite steams?

1
  • 2
    As an experiment to clear up what's happening with buffering, type (or copy-paste) a bunch of stuff into cat's input — 4096 bytes should be enough. Eventually you'll get a bunch of output all at once. Then, you won't get any more until you add even more input.
    – hobbs
    May 18, 2016 at 19:36

3 Answers 3

5

The pipes connect the output or the left command to the input of the right command. This has nothing to do with the length of the stream. However, each command in the pipeline still has it's own buffering rules. If you don't trigger them in each command you won't see them on the final output.

2
  • Then how come cat | sed '' works? I mean it should wait for the buffering limit before it is sent to stdout or not? Is it like cat sends its buffer content immediately to sed?
    – Nishant
    May 18, 2016 at 14:29
  • 1
    Buffering in a program can change based on where the output is going. This can be observed when running a command from the terminal, where stdout is most likely your screen, and compared to redirecting to a file. The former is usually line buffered, the later is usually buffered on some number of bytes, 4096 for example. May 18, 2016 at 14:38
4

That's basically a duplicate of my answer on SO. However, since nobody mentioned the stdbuf command here, I felt like I should add that here as well.

===============

Basically a process that reads from a pipe can consume the data byte by byte as soon as they are available in the pipe. However, as long as the programs are using std io functions of the libc, like read, write etc, the libc will buffer the input/output of those programs depending on whether a program is writing to a terminal or not.

By default, if a program is writing to a terminal the libc will buffer the output line wise, if it goes not to a terminal it get's buffered block wise.

On Linux, having glibc, you can influence that behaviour using the stdbuf command, like this:

stdbuf -oL cat | stdbuf -ioL sed '' | stdbuf -iL sed ''

I'm using a line based output buffer for the cat command, a line based input and output buffer for the first sed command and a line based input buffer for the last sed command.

2
  • Just to add, please notice the obvious fact there is no guarantee any arbitrary program is using the libc or any libc at all, therefore you cannot really rely on this behavior.
    – dbanet
    May 18, 2016 at 22:41
  • Yes, that's true. Programs are free to fill/flush buffers as they wish. tee for example is such a program
    – hek2mgl
    May 18, 2016 at 22:47
3

You could use the -u option of sed to minimize buffering:

cat | sed -u '' | sed ''

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .