How to use parallel with inotifywait?

Question

I currently have this:

inotifywait -m ./input -e close_write | parallel --max-args 1 "echo {1}"

For some reason this doesn't do anything at all. The output of inotifywait is the same as find (i.e. one file per line):

find ./input -type f | parallel --max-args 1 "echo {1}"

...and that works fine. Why doesn't it work with inotifywait?

Answer 1

The reason is hidden inside this statement from the documentation (man parallel)

GNU parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU parallel as input for other programs.

Reading on, you can find

--group Group output. Output from each jobs is grouped together and is only printed when the command is finished. stderr (standard error) first followed by stdout (standard output). [...] if it is acceptable that the outputs from different commands are mixed together, then disabling grouping with -u can speedup GNU parallel [...]

--group is the default. Can be reversed with -u

In your situation what's happening is that because inotifywait -m doesn't exit, parallel will collect its output forever. When inotifywait -m finished (which it never does), parallel would then output all the data it had processed.

You can "fix" this with the --ungroup (-u) flag

--ungroup | -u Ungroup output. Output is printed as soon as possible and by passes GNU parallel internal processing. This may cause output from different commands to be mixed thus should only be used if you do not care about the output.

Thus,

inotifywait ./input -e close_write | parallel --ungroup --max-args=1 echo {1}

Answer 2

https://www.gnu.org/software/parallel/parallel.html#EXAMPLE:-GNU-Parallel-as-dir-processor is pretty much what you do.

It states:

Using GNU parallel as dir processor has the same limitations as using GNU parallel as queue system/batch manager.

https://www.gnu.org/software/parallel/parallel.html#EXAMPLE:-GNU-Parallel-as-queue-system-batch-manager says:

There is a a small issue when using GNU parallel as queue system/batch manager: You have to submit JobSlot number of jobs before they will start, and after that you can submit one at a time, and job will start immediately if free slots are available. Output from the running or completed jobs are held back and will only be printed when JobSlots more jobs has been started (unless you use --ungroup or --line-buffer, in which case the output from the jobs are printed immediately). E.g. if you have 10 jobslots then the output from the first completed job will only be printed when job 11 has started, and the output of second completed job will only be printed when job 12 has started.

And this is what you are seeing.

Try this:

seq 100 | parallel --delay 1 -j1 echo | # give 1..100 one per second
  # stdout is cached by GNU Parallel, >&3 is not
  parallel -j4 'sleep 1; echo stdout {}; echo direct {} >&3; sleep 1' 3>&1