Batch Processing with one task at a time

Question

I'm used to IBM i (AS/400) batch processing with flexible queueing configuration possibilities. I'm searching a similar facility for Linux Batch processing.

Most important is that one job at a time being taken from the queue and executed. After that, the queue is scanned for more entries. If there are any, the next task will be fetched to run.

at -b comes relatively close but if a job takes so little resources that the load stays down, more tasks get executed in parallel which is not desired.

Run-parts would be another option but it is too static in behavior and doesn't run as a daemon to look for new entries in a given queue. I could run it as a regular cron job but I need to take care of jobs to delete from the queue myself.

Do I have to code my own or extend at to achieve my desired functionality or are there alternate batch queueing mechanisms available?

Answer 1

You can do this just using bash:

1- create fifo using mkfifo (you can create file too)

mkfifo mybuff

2- tail the fifo and pass each line to your executor (this will be kind of task server)

while IFS= read -r line
do
  echo "Running: '$line' with bash"
  bash -c "$line"
  echo "Finished '$line' with exit code: $?"
done < <(tail -f mybuff)

3- Send your commands to queue for example (this will client requesting tasks)

echo 'sleep 10' >> mybuff
echo 'echo "hello world"' >> mybuff
echo 'my fancy command to be executed' >> mybuff
...

PS: You can simplify step 2 like example below but this will finish task listener when you send "exit" to your command buffer

tail -f mybuff | bash -x

Answer 2

ts - task spooler. A simple unix batch system should be up for the task. It runs a task spooler/queue server and you can add a new task with a simple command like:

tsp yourcommand

You can specifify the number of slots - aka how many jobs should be executed at a time. By default this is set to one, AFAIK. It also has some advanced functionality, like running a command only if a certain previous command succeeds. You can view the queue and details for each job as well. You can find more details in the manpage.