3

I'm compiling a huge list of commands (all doing the same thing) I want executed, but as it takes a long time to compile that list, I would like execution to begin before I'm done (execution of each command typically takes longer than creating another, so there's no real risk of the list running dry).

The normal way to execute a list of commands, is to write a shell script listing the commands, but when I start execution of a script I can't add to it anymore.

The way I've found so far is to put the commands in command.list and having parallel --jobs 1 --line-buffer :::: command.list, but as it involves using parallel (I'm using GNU parallel, I don't know if it will work with the program in more-utils) for non-parallel execution of things, I think it's a bit of an abuse of parallel.

Is there a simpler way of doing it? Something that tracks which commands have been executed in case I screw up something and the list does run out, would be nice.

3
  • Can't you just do sh ./command.list or sh <(tail -n+1 -f command.list) Mar 15, 2022 at 12:09
  • sh ./command.list does not discover new commands added to the list. The solution with tail seems to do what I want, but unlike the solution with parallel it doesn't exit when the list is empty (I don't really know which I prefer). Mar 15, 2022 at 12:29
  • 1
    sh ./command.list should discover new commands added to the list as long as they are added to the list before sh has reached the end of it. You could also just do sh <(command-that-generates-that-list) or just command-that-generates-that-list | sh if the commands do not read their stdin. Mar 15, 2022 at 12:36

1 Answer 1

1

From: https://www.gnu.org/software/parallel/man.html#example-gnu-parallel-as-queue-system-batch-manager

true >jobqueue; tail -n+0 -f jobqueue | parallel --joblog my.log &
echo my_command my_arg >> jobqueue
my_job_generator >> jobqueue

This will give you a record (my.log) of which jobs have completed.

GNU Parallel version 20220222 will only output job n (and add it to my.log) when job n+1 has been added. If that is unacceptable, just add another dummy-job:

echo true dummy >> jobqueue

The behaviour is slightly different in older versions.

5
  • --joblog seems like a very valuable thing (why didn't I check the documentation to see if such a feature was implemented, parallel has so many features), and getting an answer from the author of the software makes it feel less like an abuse of the software. Mar 16, 2022 at 20:29
  • @Henriksupportsthecommunity GNU Parallel has become better by being abused: Several bugs have been uncovered by users using it in ways it was never designed to support - and fixing the bugs have improved reliability. So abuse away. :)
    – Ole Tange
    Mar 16, 2022 at 22:11
  • @OleTange it seems that using -u will make the job output available as soon as it finishes, but joblog will still be updated only after another job has started. Is there a way to make --joblog follow the same behavior? Dec 12, 2023 at 2:55
  • @LucasMenicucci -u prints the output without buffering. It does not wait until the job is done. --joblog is only updated after the job is done.
    – Ole Tange
    Dec 12, 2023 at 9:50
  • Got it, thanks! Dec 12, 2023 at 17:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .