What I want to do is something like this:
inotifywait -m ./input -e close_write -e moved_to |
parallel --max-args 1 "someSlowSingleThreadedTaskThatActsOnThisFile"
I can't imagine that that would work as-is (it doesn't seem to). I assume parallel
has to be given all of its inputs at the start, before it begins running. Therefore my question is this: Is there a way that I could add more inputs to parallel as it's already going? I imagine the inputs as a pool of work, and it just processes that pool as fast as possible, using the maximum sensible number of threads for the current system. What I want to do is add to that pool while it's still going.
How can I continuously feed parallel
with more inputs as it's running (specifically, as more files are added to the folder)? Alternatively, if parallel
isn't a good choice for this task, what is?