Suppose I have a executable which for which I want to log STDOUT and STDERR to separate files, like so:

python write.py > out 2> err

When I use this command with GNU timeout, my out file is always empty. Why does this happen, and how can I fix this?

timeout 5s python write.py > out 2> err

Example write.py:

import sys, time

i = 0
while True:    
    print i
    print >> sys.stderr, i
    i += 1
$ timeout 5s python write.py > out 2> err
$ ls -l out err
-rw-r--r-- 1 yeti yeti 10 Apr  6 08:12 err
-rw-r--r-- 1 yeti yeti  0 Apr  6 08:12 out
$ timeout 5s python -u write.py > out 2> err
$ ls -l out err
-rw-r--r-- 1 yeti yeti 10 Apr  6 08:13 err
-rw-r--r-- 1 yeti yeti 10 Apr  6 08:13 out
$ cmp err out && echo same
$ cat out 

python uses buffered writes on stdout but not on stderr. So everything written to sys.stderr is written immediately but stuff for sys.stdout is kept in a buffer until the buffer is full to minimise write operations. Within 5 seconds, the buffer does not fill enough to get written even once and timeout terminates the python interpreter.

Adding the -u option to python's invocation, writes to stdout will be unbuffered too.

You can force the same behaviour for a Python program by setting the environment variable PYTHONUNBUFFERED. For other programs you can use unbuffer from the expect package or stdbuf from GNU coreutils (see Turn off buffering in pipe).

  • So this problem actually came up in the context of a different executable (not Python), do you know a more general solution? if it helps, the executable was a make compilation using Scala's sbt. – jaynp Apr 6 '15 at 6:59
  • 2
    @dal102 unbuffer or stdbuf (see my edit) – Gilles Apr 6 '15 at 22:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.