I have a Python script that does more or less this
current_tasks = TaskManager() MAXPROCS = 8 while len(outstanding_tasks) > 0: if len(current_tasks.running) < MAXPROCS: current_tasks.addTask(outstanding_tasks.next()) else: current_tasks.wait_for_one_finish()
and outstanding_tasks.next() is basically this:
p = subprocess.Popen([task], stdout=OUTFILE, stderr=subprocess.PIPE)
waiting = True while waiting: for t in tasks: ret = t.poll() if ret not None: handle_stderr(t) waiting = False break
Fairly straightforward - spawn tasks on-demand until we're running 8 of them at a time, and then block until they finish one at a time before spawning more tasks.
The problem is this:
Each subprocess is writing is stderr to a pipe. If it crashes and it wants to write a big log message or whatever to the pipe, and that message exceeds the size of the pipe buffer, the write() will block. The process won't finish, so my controlling process will never see a return value from
poll() and go read from its stderr.
There are obviously ways around this:
- redirect stderr from my subprocesses to temporary files
- spawn a Python thread that reads from the stderr file descriptors of all running tasks and buffers them in memory
- have a select() or something in my little ad hoc event loop
But all of that is stuff I have to handle in my application code. What I want to know is: is there some way to get the behaviour of a pipe, but with a nice big elastic buffer, so that the subprocesses can always do a successful write() to their stderr and then exit, without me having to look at it until they're done?