I am a PhD student running jobs in a grid (~1500 jobs at any time). Some of the jobs go to error state which need to resubmit. The command to see the error jobs (from my local machine, in a terminal) is alien_ps -E
which gives me this:
mhaque -762153733 __ EV
mhaque -762160371 __ EV
mhaque -762376874 __ EV
.. so on..
To resubmit the jobs I need to use the alien_resubmit $jobid
(in my laptop terminal window).
So I use the following shell command:
alien_ps -E | grep 'E' | sed 's/-//' | awk '{print "alien_resubmit " $2}'
which gives me:
alien_resubmit 762153733
alien_resubmit 762160371
alien_resubmit 762376874
.. so on..
Then I copy paste this output to my terminal which then resubmit all those error jobs. Here is the real problem: I do not know when the jobs will go to error state. So I need to manually login to my laptop and check if any jobs went to error state and then resubmit it (within 24 hour or the grid kills the job).
So, I would like to write a simple shell script testmyjob.sh
which will loops this command for 5 minute intervals:
alien_ps -E | grep 'E' | sed 's/-//' | awk '{print $2}'
and see if there is any jobid
. If no job is in error then it will do the check again after 5 min (and keep doing it for definite number of times say, 200 time after I executed the shell). If it finds error jobid
then it will and run the command:
alien_resubmit jobid (..for all error jobs..).
After submitting the error jobs, it will again resume the checking for error jobs for each 5min interval.
So I will just run ./testmyjob.sh
from my laptop/desktop terminal and it will keep working in the background, and relieve me from doing the checking by myself.
I am not good at shell scripting. So please help me write a simple shell script which do this job. To be precise here is the block diagram:
(pardon my shell knowledge)
try = 0
while(try<200) do
jobid=alien_ps -E | grep 'E' | sed 's/-//' | awk '{print $2}'
if(jobid is nonzero)
alien_resubmit jobid
wait_for_5min
try+ = 1;
end do
Please note there are always several jobs in error state (see the examples I gave) and I do not know how you will assign the jobs to one variable or make an array? Any kind of help with script is very much appreciated.