6

I'm used to IBM i (AS/400) batch processing with flexible queueing configuration possibilities. I'm searching a similar facility for Linux Batch processing.

Most important is that one job at a time being taken from the queue and executed. After that, the queue is scanned for more entries. If there are any, the next task will be fetched to run.

at -b comes relatively close but if a job takes so little resources that the load stays down, more tasks get executed in parallel which is not desired.

Run-parts would be another option but it is too static in behavior and doesn't run as a daemon to look for new entries in a given queue. I could run it as a regular cron job but I need to take care of jobs to delete from the queue myself.

Do I have to code my own or extend at to achieve my desired functionality or are there alternate batch queueing mechanisms available?

3
  • Have you looked at ts/tsp? It has some sort of task spooler server and allows you to run commands after another and should offer some advanced functionality.
    – rudib
    Commented Dec 28, 2019 at 18:52
  • Also see: manpages.ubuntu.com/manpages/xenial/man1/tsp.1.html
    – rudib
    Commented Dec 28, 2019 at 18:54
  • @rudib Thanks a lot! Ts seems to match closely what I need. If you move your comment to an answer, I can mark this as done.
    – PoC
    Commented Dec 29, 2019 at 19:43

2 Answers 2

4

You can do this just using bash:

1- create fifo using mkfifo (you can create file too)

mkfifo mybuff

2- tail the fifo and pass each line to your executor (this will be kind of task server)

while IFS= read -r line
do
  echo "Running: '$line' with bash"
  bash -c "$line"
  echo "Finished '$line' with exit code: $?"
done < <(tail -f mybuff)

3- Send your commands to queue for example (this will client requesting tasks)

echo 'sleep 10' >> mybuff
echo 'echo "hello world"' >> mybuff
echo 'my fancy command to be executed' >> mybuff
...

PS: You can simplify step 2 like example below but this will finish task listener when you send "exit" to your command buffer

tail -f mybuff | bash -x
2
  • I get an error Syntax error: redirection unexpected on line 6 of the "server" part
    – kostrykin
    Commented Aug 20, 2020 at 0:53
  • @theV0ID you can use bidirectional pipe for old versions of shell: tail -f mybuff | while read LINE; do bash -c "$LINE"; done
    – Abdurrahim
    Commented Aug 20, 2020 at 17:05
3

ts - task spooler. A simple unix batch system should be up for the task. It runs a task spooler/queue server and you can add a new task with a simple command like:

tsp yourcommand

You can specifify the number of slots - aka how many jobs should be executed at a time. By default this is set to one, AFAIK. It also has some advanced functionality, like running a command only if a certain previous command succeeds. You can view the queue and details for each job as well. You can find more details in the manpage.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .