6

I'm used to IBM i (AS/400) batch processing with flexible queueing configuration possibilities. I'm searching a similar facility for Linux Batch processing.

Most important is that one job at a time being taken from the queue and executed. After that, the queue is scanned for more entries. If there are any, the next task will be fetched to run.

at -b comes relatively close but if a job takes so little resources that the load stays down, more tasks get executed in parallel which is not desired.

Run-parts would be another option but it is too static in behavior and doesn't run as a daemon to look for new entries in a given queue. I could run it as a regular cron job but I need to take care of jobs to delete from the queue myself.

Do I have to code my own or extend at to achieve my desired functionality or are there alternate batch queueing mechanisms available?

3
  • Have you looked at ts/tsp? It has some sort of task spooler server and allows you to run commands after another and should offer some advanced functionality. – rudib Dec 28 '19 at 18:52
  • Also see: manpages.ubuntu.com/manpages/xenial/man1/tsp.1.html – rudib Dec 28 '19 at 18:54
  • @rudib Thanks a lot! Ts seems to match closely what I need. If you move your comment to an answer, I can mark this as done. – PoC Dec 29 '19 at 19:43
3

ts - task spooler. A simple unix batch system should be up for the task. It runs a task spooler/queue server and you can add a new task with a simple command like:

tsp yourcommand

You can specifify the number of slots - aka how many jobs should be executed at a time. By default this is set to one, AFAIK. It also has some advanced functionality, like running a command only if a certain previous command succeeds. You can view the queue and details for each job as well. You can find more details in the manpage.

4

You can do this just using bash:

1- create fifo using mkfifo (you can create file too)

mkfifo mybuff

2- tail the fifo and pass each line to your executor (this will be kind of task server)

while IFS= read -r line
do
  echo "Running: '$line' with bash"
  bash -c "$line"
  echo "Finished '$line' with exit code: $?"
done < <(tail -f mybuff)

3- Send your commands to queue for example (this will client requesting tasks)

echo 'sleep 10' >> mybuff
echo 'echo "hello world"' >> mybuff
echo 'my fancy command to be executed' >> mybuff
...

PS: You can simplify step 2 like example below but this will finish task listener when you send "exit" to your command buffer

tail -f mybuff | bash -x
2
  • I get an error Syntax error: redirection unexpected on line 6 of the "server" part – theV0ID Aug 20 '20 at 0:53
  • @theV0ID you can use bidirectional pipe for old versions of shell: tail -f mybuff | while read LINE; do bash -c "$LINE"; done – Abdurrahim Aug 20 '20 at 17:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.