How to write script for checking Error jobs and auto resubmit?

Question

I am a PhD student running jobs in a grid (~1500 jobs at any time). Some of the jobs go to error state which need to resubmit. The command to see the error jobs (from my local machine, in a terminal) is alien_ps -E which gives me this:

   mhaque -762153733  __  EV                                   
   mhaque -762160371  __  EV                                   
   mhaque -762376874  __  EV 

.. so on..

To resubmit the jobs I need to use the alien_resubmit $jobid (in my laptop terminal window). So I use the following shell command:

alien_ps -E | grep 'E' | sed 's/-//' | awk '{print "alien_resubmit " $2}' 

which gives me:

alien_resubmit 762153733
alien_resubmit 762160371
alien_resubmit 762376874

.. so on..

Then I copy paste this output to my terminal which then resubmit all those error jobs. Here is the real problem: I do not know when the jobs will go to error state. So I need to manually login to my laptop and check if any jobs went to error state and then resubmit it (within 24 hour or the grid kills the job). So, I would like to write a simple shell script testmyjob.sh which will loops this command for 5 minute intervals:

alien_ps -E | grep 'E' | sed 's/-//' | awk '{print $2}'

and see if there is any jobid. If no job is in error then it will do the check again after 5 min (and keep doing it for definite number of times say, 200 time after I executed the shell). If it finds error jobid then it will and run the command:

alien_resubmit jobid         (..for all error jobs..).

After submitting the error jobs, it will again resume the checking for error jobs for each 5min interval.

So I will just run ./testmyjob.sh from my laptop/desktop terminal and it will keep working in the background, and relieve me from doing the checking by myself. I am not good at shell scripting. So please help me write a simple shell script which do this job. To be precise here is the block diagram: (pardon my shell knowledge)

try = 0
while(try<200) do
jobid=alien_ps -E | grep 'E' | sed 's/-//' | awk '{print $2}'
if(jobid is nonzero) 
alien_resubmit jobid   
wait_for_5min
try+ = 1;
end do

Please note there are always several jobs in error state (see the examples I gave) and I do not know how you will assign the jobs to one variable or make an array? Any kind of help with script is very much appreciated.

Answer 1

You can save this script in /usr/local/bin/resubmit_jobs

#!/bin/bash
for job in $(alien_ps -E | grep 'E' | awk '{print $2}' | sed 's/-//'); do
  alien_resubmit $job
done

Then you should set the script as executable by running this command:

chmod a+x /usr/local/bin/resubmit_jobs

Then add a entry to your crontab. Just type: crontab -e and add those lines:

MAILTO=your.email@ddresse.com # so you'll get the output of your cron jobs by mail
*/5 * * * * /usr/local/bin/resubmit_jobs # this add a cron jobs that will run your resubmit_jobs script every 5 minutes

If you want to run this script manually you just have to make: resubmit_jobs

Answer 2

iterations=200
while [[ $iterations -gt 1 ]]; do
    for job in $(alien_ps -E | grep 'E' | awk '{print $2}' | sed 's/-//'); do
        alien_resubmit $job
    done
    sleep 300
    iterations=$(($iterations-1))
done