1

I have a process like this that generates a predefined number of files but at random intervals:

#!/bin/bash

for i in {1..10}
do
  sleep $(shuf -i 20-60 -n 1)
  echo $i > file_$i.txt
done

I have another process that runs on each of those files independently using GNU Parallel like so:

parallel wc -l ::: file_{1..10}.txt

As expected, Parallel runs on the files that are currently available. Is there a way to make Parallel wait for the remaining files to be available and run as soon as they are?

2 Answers 2

1

Look at https://www.gnu.org/software/parallel/parallel_examples.html#example-gnu-parallel-as-queue-system-batch-manager

Terminal 1:

true >jobqueue; tail -n+0 -f jobqueue | parallel -u

(-u is needed if you want output on screen immediately. Otherwise the output will be delayed until the next job completes. In both cases the job is run immediately).

Terminal 2:

#!/bin/bash

for i in {1..10}
do
  sleep $(shuf -i 20-60 -n 1)
  echo $i > file_$i.txt
  echo file_$i.txt >> jobqueue
done

If the files are the only files created in my_dir look at https://www.gnu.org/software/parallel/parallel_examples.html#example-gnu-parallel-as-dir-processor

inotifywait -qmre MOVED_TO -e CLOSE_WRITE --format %w%f my_dir |
  parallel -u echo

This way you do not need the jobqueue file.

0
1

If the files are always created in order from 1 to 10, you could just test for the existence of file_10.txt. e.g.

   until [ -e file_10.txt ] ; do sleep 20 ; done
   parallel wc -l ::: file_{1..10}.txt

Otherwise, you could count the number of matching files, e.g.

numfiles=$(find . -name 'file*.txt' | wc -l)
while [ "$numfiles" -lt 10 ] ; do
  sleep 20
  numfiles=$(find . -name 'file*.txt' | wc -l)
done
parallel wc -l ::: file_{1..10}.txt

(note: piping find to wc -l won't work if any of the filenames contain newlines because wc -l counts the number of lines)

I've used a sleep time of 20 seconds because your for loop sleeps for a minimum of 20 seconds. Adjust the sleep time to suit your needs - increase it if the file creation process takes a long time. If it's really fast, reduce it.

3
  • Thanks for the answer. However, this will mean to wait for all files. I am trying to find a way to get the Parallel run as the files become available.
    – Ketan
    Commented Aug 2, 2023 at 14:46
  • OK, that sounds more like a queue. You could implement one using two arrays, one to hold the current files you're about to pass to parallel, and one to hold the ones already processed. e.g. using something like (untested) mapfile -t current < <(find . -name 'file*.txt' | grep -v -F -f <(printf '%s\n' "${seen[@]}")) and, later, seen+=( "${current[@]}" ). This can get complicated if the queue files aren't being deleted as they're processed, and even more complicated if new files are added to the queue that have the same name as a file that has already been processed.
    – cas
    Commented Aug 2, 2023 at 15:10
  • that should have said "deleted or moved out of the queue dir as they're processed".
    – cas
    Commented Aug 2, 2023 at 15:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .