6

Suppose I have a executable which for which I want to log STDOUT and STDERR to separate files, like so:

python write.py > out 2> err

When I use this command with GNU timeout, my out file is always empty. Why does this happen, and how can I fix this?

timeout 5s python write.py > out 2> err

Example write.py:

#!/bin/bash
import sys, time

i = 0
while True:    
    print i
    print >> sys.stderr, i
    time.sleep(1)
    i += 1

1 Answer 1

7
$ timeout 5s python write.py > out 2> err
$ ls -l out err
-rw-r--r-- 1 yeti yeti 10 Apr  6 08:12 err
-rw-r--r-- 1 yeti yeti  0 Apr  6 08:12 out
$ timeout 5s python -u write.py > out 2> err
$ ls -l out err
-rw-r--r-- 1 yeti yeti 10 Apr  6 08:13 err
-rw-r--r-- 1 yeti yeti 10 Apr  6 08:13 out
$ cmp err out && echo same
same
$ cat out 
0
1
2
3
4

python uses buffered writes on stdout but not on stderr. So everything written to sys.stderr is written immediately but stuff for sys.stdout is kept in a buffer until the buffer is full to minimise write operations. Within 5 seconds, the buffer does not fill enough to get written even once and timeout terminates the python interpreter.

Adding the -u option to python's invocation, writes to stdout will be unbuffered too.

You can force the same behaviour for a Python program by setting the environment variable PYTHONUNBUFFERED. For other programs you can use unbuffer from the expect package or stdbuf from GNU coreutils (see Turn off buffering in pipe).

2
  • So this problem actually came up in the context of a different executable (not Python), do you know a more general solution? if it helps, the executable was a make compilation using Scala's sbt.
    – jaynp
    Apr 6, 2015 at 6:59
  • 2
    @dal102 unbuffer or stdbuf (see my edit) Apr 6, 2015 at 22:51

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .