Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a Python script that does more or less this

current_tasks = TaskManager()
MAXPROCS = 8

while len(outstanding_tasks) > 0:
    if len(current_tasks.running) < MAXPROCS:
      current_tasks.addTask(outstanding_tasks.next())
    else:
      current_tasks.wait_for_one_finish()

and outstanding_tasks.next() is basically this:

p = subprocess.Popen([task], stdout=OUTFILE, stderr=subprocess.PIPE)

and current_tasks.wait_for_one_finish():

waiting = True
while waiting:
    for t in tasks:
        ret = t.poll()
        if ret not None:
            handle_stderr(t)
            waiting = False
            break

Fairly straightforward - spawn tasks on-demand until we're running 8 of them at a time, and then block until they finish one at a time before spawning more tasks.

The problem is this:

stderr=subprocess.PIPE

Each subprocess is writing is stderr to a pipe. If it crashes and it wants to write a big log message or whatever to the pipe, and that message exceeds the size of the pipe buffer, the write() will block. The process won't finish, so my controlling process will never see a return value from poll() and go read from its stderr.

There are obviously ways around this:

  • redirect stderr from my subprocesses to temporary files
  • spawn a Python thread that reads from the stderr file descriptors of all running tasks and buffers them in memory
  • have a select() or something in my little ad hoc event loop

But all of that is stuff I have to handle in my application code. What I want to know is: is there some way to get the behaviour of a pipe, but with a nice big elastic buffer, so that the subprocesses can always do a successful write() to their stderr and then exit, without me having to look at it until they're done?

share|improve this question
    
You can't look at the processes retcode instead of parsing it's stderr (which is generally bad for error control)? –  jordanm Feb 9 '13 at 3:30
    
I'm looking at the retcode to decide whether it failed, but I need the stderr to get the log output and send it off or propagate the error reporting. –  Cerales Feb 9 '13 at 4:19
add comment

1 Answer 1

up vote 1 down vote accepted

The short answer is: there isn't.

You've already highlighted the workarounds necessary to deal with large data being sent through a subprocess pipe. The "nice big elastic buffer" pipe doesn't exist. This is called out in the Subprocess Management Python documentation as a potential source of deadlocks, with the added solution that you can call proc.communicate() to read from stderr. The problem in your case is that you can't call communicate() on all processes at the same time, and that method blocks until all data is read.

If it were I, I would probably use a select() call on all of the stderr processes rather than a proc.poll() loop. select() can block until any process does something, and when the process exits, it will close the stderr pipe, so you kill two birds with one stone (know when data is written to stderr and know when the process the dies).

share|improve this answer
    
Thanks, a select() loop on stderr is actually perfect for what I'm trying to do - as long as I make sure I don't close stderr in the child processes before they exit. –  Cerales Feb 20 '13 at 23:00
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.