I have the following problem. I have a machine learning process that runs in the background. Even better, I have plenty (let's say 5) that I want to run them. Each of them is CPU and GPU power consuming and therefore would like to run only one of them at the time for a specific period of time (i.e. 1h, actually the script itself is able to close after 1h), then stop it and run another one. This way I may train all of them while the machine is running for like a several days simultaneously.
The challenge is as follows: I want to run them balanced/rotated. Instead of running the first one, then second, third etc. I want the script/handler to ensure that after first one is run and machine is restarted (or upstart service is restarted or something like that) then second is run. I want to avoid situation that after every restart it starts with the same process and most of the time is spent on the first one, and least on the last one.
Could anyone suggest an out-of-the-box solution (except for writing my own script/handler which is obviously easy but I would like something thoroughly tested instead of something of my own that I need to manage).