I have a task that processes a list of files on stdin. The start-up time of the program is substantial, and the amount of time each file takes varies widely. I want to spawn a substantial number of these processes, then dispatch work to whichever ones are not busy. There are several different commandline tools that almost do what I want, I've narrowed it down to two almost working options:
find . -type f | split -n r/24 -u --filter="myjob" find . -type f | parallel --pipe -u -l 1 myjob
The problem is that
split does a pure round-robin, so one of the processes gets behind and stays behind, delaying the completion of the entire operation; while
parallel wants to spawn one process per N lines or bytes of input and I wind up spending way too much time on startup overhead.
Is there something like this that will re-use the processes and feed lines to whichever processes have unblocked stdins?