2

I currently have this:

inotifywait -m ./input -e close_write | parallel --max-args 1 "echo {1}"

For some reason this doesn't do anything at all. The output of inotifywait is the same as find (i.e. one file per line):

find ./input -type f | parallel --max-args 1 "echo {1}"

...and that works fine. Why doesn't it work with inotifywait?

2
  • I did, they both output one line of text for file found.
    – Clonkex
    Feb 21, 2021 at 23:53
  • I would use find ./inpuf -type f -print0 | parallel --null … to keep the data clean.
    – Larry
    Jun 10, 2021 at 22:01

2 Answers 2

1

The reason is hidden inside this statement from the documentation (man parallel)

GNU parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU parallel as input for other programs.

Reading on, you can find

--group Group output. Output from each jobs is grouped together and is only printed when the command is finished. stderr (standard error) first followed by stdout (standard output). [...] if it is acceptable that the outputs from different commands are mixed together, then disabling grouping with -u can speedup GNU parallel [...]

--group is the default. Can be reversed with -u

In your situation what's happening is that because inotifywait -m doesn't exit, parallel will collect its output forever. When inotifywait -m finished (which it never does), parallel would then output all the data it had processed.

You can "fix" this with the --ungroup (-u) flag

--ungroup | -u Ungroup output. Output is printed as soon as possible and by passes GNU parallel internal processing. This may cause output from different commands to be mixed thus should only be used if you do not care about the output.

Thus,

inotifywait ./input -e close_write | parallel --ungroup --max-args=1 echo {1}
1
  • Sorry - this answer is wrong and misleading. The job is ‘echo’, not inotifywait. Inotifywait is just the generator of data that feeds into parallel, just like find. You don't need to ungroup output (see Ole Tang's answer). I think the main reason Clonkex was seeing a problem was due to max-args and {1} - I don't think that's what you want in this case.
    – Larry
    Jun 10, 2021 at 22:08
1

https://www.gnu.org/software/parallel/parallel.html#EXAMPLE:-GNU-Parallel-as-dir-processor is pretty much what you do.

It states:

Using GNU parallel as dir processor has the same limitations as using GNU parallel as queue system/batch manager.

https://www.gnu.org/software/parallel/parallel.html#EXAMPLE:-GNU-Parallel-as-queue-system-batch-manager says:

There is a a small issue when using GNU parallel as queue system/batch manager: You have to submit JobSlot number of jobs before they will start, and after that you can submit one at a time, and job will start immediately if free slots are available. Output from the running or completed jobs are held back and will only be printed when JobSlots more jobs has been started (unless you use --ungroup or --line-buffer, in which case the output from the jobs are printed immediately). E.g. if you have 10 jobslots then the output from the first completed job will only be printed when job 11 has started, and the output of second completed job will only be printed when job 12 has started.

And this is what you are seeing.

Try this:

seq 100 | parallel --delay 1 -j1 echo | # give 1..100 one per second
  # stdout is cached by GNU Parallel, >&3 is not
  parallel -j4 'sleep 1; echo stdout {}; echo direct {} >&3; sleep 1' 3>&1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.