I have a Python script that does more or less this

current_tasks = TaskManager()

while len(outstanding_tasks) > 0:
    if len(current_tasks.running) < MAXPROCS:

and outstanding_tasks.next() is basically this:

p = subprocess.Popen([task], stdout=OUTFILE, stderr=subprocess.PIPE)

and current_tasks.wait_for_one_finish():

waiting = True
while waiting:
    for t in tasks:
        ret = t.poll()
        if ret not None:
            waiting = False

Fairly straightforward - spawn tasks on-demand until we're running 8 of them at a time, and then block until they finish one at a time before spawning more tasks.

The problem is this:


Each subprocess is writing is stderr to a pipe. If it crashes and it wants to write a big log message or whatever to the pipe, and that message exceeds the size of the pipe buffer, the write() will block. The process won't finish, so my controlling process will never see a return value from poll() and go read from its stderr.

There are obviously ways around this:

  • redirect stderr from my subprocesses to temporary files
  • spawn a Python thread that reads from the stderr file descriptors of all running tasks and buffers them in memory
  • have a select() or something in my little ad hoc event loop

But all of that is stuff I have to handle in my application code. What I want to know is: is there some way to get the behaviour of a pipe, but with a nice big elastic buffer, so that the subprocesses can always do a successful write() to their stderr and then exit, without me having to look at it until they're done?

  • You can't look at the processes retcode instead of parsing it's stderr (which is generally bad for error control)? – jordanm Feb 9 '13 at 3:30
  • I'm looking at the retcode to decide whether it failed, but I need the stderr to get the log output and send it off or propagate the error reporting. – Cera Feb 9 '13 at 4:19

The short answer is: there isn't.

You've already highlighted the workarounds necessary to deal with large data being sent through a subprocess pipe. The "nice big elastic buffer" pipe doesn't exist. This is called out in the Subprocess Management Python documentation as a potential source of deadlocks, with the added solution that you can call proc.communicate() to read from stderr. The problem in your case is that you can't call communicate() on all processes at the same time, and that method blocks until all data is read.

If it were I, I would probably use a select() call on all of the stderr processes rather than a proc.poll() loop. select() can block until any process does something, and when the process exits, it will close the stderr pipe, so you kill two birds with one stone (know when data is written to stderr and know when the process the dies).

  • Thanks, a select() loop on stderr is actually perfect for what I'm trying to do - as long as I make sure I don't close stderr in the child processes before they exit. – Cera Feb 20 '13 at 23:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.