Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a script which calls two commands:

long_running_command | print_progress

The long_running_command prints a progress but I'm unhappy with it. I'm using print_progress to make it more nice (namely, I print the progress in a single line).

The problem: The pipe activates a 4K buffer, to the nice print program gets nothing ... nothing ... nothing ... a whole lot ... :)

How can I disable the 4K buffer for the long_running_command (no, I don't have the source)?

share|improve this question

migrated from stackoverflow.com Nov 25 '11 at 22:03

This question came from our site for professional and enthusiast programmers.

1  
So when you run long_running_command without piping you can see the progress updates properly, but when piping they get buffered? –  second Jun 16 '09 at 10:58
    
Yes, that's exactly what happens. –  Aaron Digulla Jun 16 '09 at 11:50
10  
The inability for a simple way of controlling buffering has been a problem for decades. For example, see: marc.info/?l=glibc-bug&m=98313957306297&w=4 which basicly says "I can't be arsed doing this and here's some clap-trap to justify my position" –  Adrian Pronk Oct 19 '10 at 21:59

10 Answers 10

up vote 74 down vote accepted

You can use the expect command unbuffer, e.g.

unbuffer long_running_command | print_progress

unbuffer connects to long_running_command via a pseudoterminal (pty), which makes the system treat it as an interactive process, therefore not using the 4-kiB buffering in the pipeline that is the likely cause of the delay.

For longer pipelines, you may have to unbuffer each command (except the final one), e.g.

unbuffer x | unbuffer -p y | z
share|improve this answer
3  
In fact, the use of a pty to connect to interactive processes is true of expect in general. –  cheduardo Jun 17 '09 at 7:58
1  
So fscking simple ! I am always amazed when I discover this kind of utilities. I knew expect, but not unbuffer –  shodanex Jun 17 '09 at 15:36
8  
When pipelining calls to unbuffer, you should use the -p argument so that unbuffer reads from stdin. –  Chris Conway Oct 6 '09 at 20:18
13  
Note: On debian systems, this is called expect_unbuffer and is in the expect-dev package, not the expect package –  bdonlan Jan 24 '11 at 11:14
3  
The link for "unbuffer" is dead –  qdii May 12 '13 at 11:38

Another way to skin this cat is to use the stdbuf program, which is part of the GNU Coreutils.

stdbuf -i0 -o0 -e0 command

This turns off buffering completely for input, output and error. For some applications, line buffering may be more suitable for performance reasons:

stdbuf -oL -eL command
share|improve this answer
9  
Why doesn’t this get more upvotes? it's a better solution imho. Does it have any downsides? –  qdii May 12 '13 at 11:39
2  
"unbuffer" needs to be installed in Ubuntu, which is inside the package: expect-dev which is 2MB... –  lepe Jun 27 '13 at 6:21
1  
This works great on the default raspbian install to unbuffer logging. I found sudo stdbuff … command works although stdbuff … sudo command didn't. –  natevw Jul 10 '13 at 6:05
    
@qdii stdbuf does not work with tee, because tee overwrites the defaults set by stdbuf. See the manual page of stdbuf. –  ceving Jun 30 at 11:51
    
@lepe Bizarrely, unbuffer has dependencies on x11 and tcl/tk, meaning it actually needs >80 MB if you're installing it on a server without them. –  jpatokal Aug 28 at 12:27

If it is a problem with the libc modifying its buffering / flushing when output does not go to a terminal, you should try socat. You can create a bidirectional stream between almost any kind of I/O mechanism. One of those is a forked program speaking to a pseudo tty.

 socat EXEC:long_running_command,pty,ctty STDIO 

What it does is

  • create a pseudo tty
  • fork long_running_command with the slave side of the pty as stdin/stdout
  • establish a bidirectional stream between the master side of the pty and the second address (here it is STDIO)

If this gives you the same output as long_running_command, then you can continue with a pipe.

Edit : Wow Did not see the unbuffer answer ! Well, socat is a great tool anyway, so I might just leave this answer

share|improve this answer
    
...and I didn't know about socat - looks kinda like netcat only perhaps more so. ;) Thanks and +1. –  cheduardo Jun 20 '09 at 9:32
    
+1 for explaining the reason behind the buffering! –  musiphil Feb 8 at 7:55

Yet another way to turn on line-buffering output mode for the long_running_command is to use the script command that runs your long_running_command in a pseudo terminal (pty).

script -q /dev/null long_running_command | print_progress      # FreeBSD, Mac OS X
script -c "long_running_command" /dev/null | print_progress    # Linux
share|improve this answer
3  
+1 nice trick, since script is such an old command, it should be available on all Unix-like platforms. –  Aaron Digulla Jan 20 '13 at 13:01
    
you also need -q on Linux: script -q -c 'long_running_command' /dev/null | print_progress –  J.F. Sebastian Apr 11 '13 at 12:51
    
It seems like script reads from stdin, which makes it impossible to run such a long_running_command in the background, at least when started from interactive terminal. To workaround, I was able to redirect stdin from /dev/null, since my long_running_command doesn't use stdin. –  haridsv Nov 15 '13 at 12:44
    
Even works on Android. –  user1147688 Jul 2 at 23:36

For grep, sed and awk you can force output to be line buffered. You can use:

grep --line-buffered

Force output to be line buffered.  By default, output is line buffered when standard output is a terminal and block buffered other-wise.

sed -u

Make output line buffered.

See this page for more information: http://www.perkin.org.uk/posts/how-to-fix-stdio-buffering.html

share|improve this answer

It used to be the case, and probably still is the case, that when standard output is written to a terminal, it is line buffered by default - when a newline is written, the line is written to the terminal. When standard output is sent to a pipe, it is fully buffered - so the data is only sent to the next process in the pipeline when the standard I/O buffer is filled.

That's the source of the trouble. I'm not sure whether there is much you can do to fix it without modifying the program writing into the pipe. You could use the setvbuf() function with the _IOLBF flag to unconditionally put stdout into line buffered mode. But I don't see an easy way to enforce that on a program. Or the program can do fflush() at appropriate points (after each line of output), but the same comment applies.

I suppose that if you replaced the pipe with a pseudo-terminal, then the standard I/O library would think the output was a terminal (because it is a type of terminal) and would line buffer automatically. That is a complex way of dealing with things, though.

share|improve this answer

You can use

long_running_command 1>&2 |& print_progress

The problem is that libc will line-buffer when stdout to screen, and full-buffer when stdout to a file. But no-buffer for stderr.

I don't think it's the problem with pipe buffer, it's all about libc's buffer policy.

share|improve this answer
    
You're right; my question is still: How can I influence libc's buffer policy without recompiling? –  Aaron Digulla Apr 4 at 8:55

I don't think the problem is with the pipe. It sounds like your long running process is not flushing its own buffer frequently enough. Changing the pipe's buffer size would be a hack to get round it, but I don't think its possible without rebuilding the kernel - something you wouldn't want to do as a hack, as it probably aversley affect a lot of other processes.

share|improve this answer
10  
The root cause is that libc switches to 4k buffering if the stdout is not a tty. –  Aaron Digulla Jun 16 '09 at 11:50
4  
That is very interesting ! because pipe don't cause any buffering. They provide buffering, but if you read from a pipe, you get whatever data is available, you don't have to wait for a buffer in the pipe. So the culprit would be the stdio buffering in the application. –  shodanex Jun 16 '09 at 13:58

According to this the pipe buffer size seems to be set in the kernel and would require you to recompile your kernel to alter.

share|improve this answer
4  
I believe that is a different buffer. –  Samuel Edwin Ward Jan 8 '13 at 21:58

According to this post here, you could try reducing the pipe ulimit to one single 512-byte block. It certainly won't turn off buffering, but well, 512 bytes is way less than 4K :3

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.