Adding to while executing a list of commands

Question

I'm compiling a huge list of commands (all doing the same thing) I want executed, but as it takes a long time to compile that list, I would like execution to begin before I'm done (execution of each command typically takes longer than creating another, so there's no real risk of the list running dry).

The normal way to execute a list of commands, is to write a shell script listing the commands, but when I start execution of a script I can't add to it anymore.

The way I've found so far is to put the commands in command.list and having parallel --jobs 1 --line-buffer :::: command.list, but as it involves using parallel (I'm using GNU parallel, I don't know if it will work with the program in more-utils) for non-parallel execution of things, I think it's a bit of an abuse of parallel.

Is there a simpler way of doing it? Something that tracks which commands have been executed in case I screw up something and the list does run out, would be nice.

Answer 1

From: https://www.gnu.org/software/parallel/man.html#example-gnu-parallel-as-queue-system-batch-manager

true >jobqueue; tail -n+0 -f jobqueue | parallel --joblog my.log &
echo my_command my_arg >> jobqueue
my_job_generator >> jobqueue

This will give you a record (my.log) of which jobs have completed.

GNU Parallel version 20220222 will only output job n (and add it to my.log) when job n+1 has been added. If that is unacceptable, just add another dummy-job:

echo true dummy >> jobqueue

The behaviour is slightly different in older versions.