Daemon service that runs a sequence of processes (balanced distribution)

Question

I have the following problem. I have a machine learning process that runs in the background. Even better, I have plenty (let's say 5) that I want to run them. Each of them is CPU and GPU power consuming and therefore would like to run only one of them at the time for a specific period of time (i.e. 1h, actually the script itself is able to close after 1h), then stop it and run another one. This way I may train all of them while the machine is running for like a several days simultaneously.

The challenge is as follows: I want to run them balanced/rotated. Instead of running the first one, then second, third etc. I want the script/handler to ensure that after first one is run and machine is restarted (or upstart service is restarted or something like that) then second is run. I want to avoid situation that after every restart it starts with the same process and most of the time is spent on the first one, and least on the last one.

Could anyone suggest an out-of-the-box solution (except for writing my own script/handler which is obviously easy but I would like something thoroughly tested instead of something of my own that I need to manage).

Answer 1

It depends how robust you want this to be. A simple-minded solution that runs the "next" program in a circular list might look like this:

#! /usr/bin/env bash

programs=(program1 program2 program3 program4 program5)
state=$HOME/.${0#*/}_state

if [ -f "$state" ]; then
    read idx <"$state"
    if expr "$idx" : '[0-9][0-9]*$' >/dev/null; then
        let idx=(idx+1)%${#programs[@]}
    else
        let idx=0
    fi
else
    let idx=0
fi
printf '%d\n' "$idx" >"$state" || exit 1

exec "${programs[$idx]}" "$@"

If you want something more robust you need locking, and probably keep state in a database.