Write Python stdout to file immediately

Question

When trying to write the stdout from a Python script to a text file (python script.py > log), the text file is created when the command is started, but the actual content isn't written until the Python script finishes. For example:

script.py:

import time
for i in range(10):
    print('bla')
    time.sleep(5)

prints to stdout every 5 seconds when called with python script.py, but when I call python script.py > log, the size of the log file stays zero until the script finishes. Is it possible to directly write to the log file, such that you can follow the progress of the script (e.g. using tail)?

EDIT It turns out that python -u script.py does the trick, I didn't know about the buffering of stdout.

Answer 1

This is happening because normally when process STDOUT is redirected to something other than a terminal, then the output is buffered into some OS-specific-sized buffer (perhaps 4k or 8k in many cases). Conversely, when outputting to a terminal, STDOUT will be line-buffered or not buffered at all, so you'll see output after each \n or for each character.

You can generally change the STDOUT buffering with the stdbuf utility:

stdbuf -oL python script.py > log

Now if you tail -F log, you should see each line output immediately as it is generated.

---

Alternatively explicit flushing of the output stream after each print should achieve the same. It looks like sys.stdout.flush() should achieve this in Python. If you are using Python 3.3 or newer, the print function also has a flush keyword that does this: print('hello', flush=True).

Answer 2

This should do the job:

import time, sys
for i in range(10):
    print('bla')
    sys.stdout.flush()
    time.sleep(5)

As Python will buffer the stdout by default, here i have used sys.stdout.flush() to flush the buffer.

Another solution would be to use the -u(unbuffered) switch of python. So, the following will do too:

python -u script.py >> log

Answer 3

Variation on the theme of using python's own option for unbuffered output would be to use #!/usr/bin/python -u as first line.

With #!/usr/bin/env python that extra argument not gonna work, so alternatively,one could run PYTHONUNBUFFERED=1 ./my_scriipt.py > output.txt or do it in two steps:

$ export PYTHONUNBUFFERED=1
$ ./myscript.py

Answer 4

You should pass flush=True to the print function:

import time

for i in range(10):
    print('bla', flush=True)
    time.sleep(5)

According to the documentation, by default, print doesn't enforce anything about flushing:

Whether output is buffered is usually determined by file, but if the flush keyword argument is true, the stream is forcibly flushed.

And the documentation for sys's strems says:

When interactive, standard streams are line-buffered. Otherwise, they are block-buffered like regular text files. You can override this value with the -u command-line option.

---

If you are stuck with an ancient version of python you have to call the flush method of the sys.stdout stream:

import sys
import time

for i in range(10):
    print('bla')
    sys.stdout.flush()
    time.sleep(5)