I am a PhD student running jobs in a grid (~1500 jobs at any time). Some of the jobs go to error state which need to resubmit. The command to see the error jobs (from my local machine, in a terminal) is alien_ps -E which gives me this:

   mhaque -762153733  __  EV                                   
   mhaque -762160371  __  EV                                   
   mhaque -762376874  __  EV 

.. so on..

To resubmit the jobs I need to use the alien_resubmit $jobid (in my laptop terminal window). So I use the following shell command:

alien_ps -E | grep 'E' | sed 's/-//' | awk '{print "alien_resubmit " $2}' 

which gives me:

alien_resubmit 762153733
alien_resubmit 762160371
alien_resubmit 762376874

.. so on..

Then I copy paste this output to my terminal which then resubmit all those error jobs. Here is the real problem: I do not know when the jobs will go to error state. So I need to manually login to my laptop and check if any jobs went to error state and then resubmit it (within 24 hour or the grid kills the job). So, I would like to write a simple shell script testmyjob.sh which will loops this command for 5 minute intervals:

alien_ps -E | grep 'E' | sed 's/-//' | awk '{print $2}'

and see if there is any jobid. If no job is in error then it will do the check again after 5 min (and keep doing it for definite number of times say, 200 time after I executed the shell). If it finds error jobid then it will and run the command:

alien_resubmit jobid         (..for all error jobs..).

After submitting the error jobs, it will again resume the checking for error jobs for each 5min interval.

So I will just run ./testmyjob.sh from my laptop/desktop terminal and it will keep working in the background, and relieve me from doing the checking by myself. I am not good at shell scripting. So please help me write a simple shell script which do this job. To be precise here is the block diagram: (pardon my shell knowledge)

try = 0
while(try<200) do
jobid=alien_ps -E | grep 'E' | sed 's/-//' | awk '{print $2}'
if(jobid is nonzero) 
alien_resubmit jobid   
wait_for_5min
try+ = 1;
end do

Please note there are always several jobs in error state (see the examples I gave) and I do not know how you will assign the jobs to one variable or make an array? Any kind of help with script is very much appreciated.

You can save this script in /usr/local/bin/resubmit_jobs

#!/bin/bash
for job in $(alien_ps -E | grep 'E' | awk '{print $2}' | sed 's/-//'); do
  alien_resubmit $job
done

Then you should set the script as executable by running this command:

chmod a+x /usr/local/bin/resubmit_jobs

Then add a entry to your crontab. Just type: crontab -e and add those lines:

MAILTO=your.email@ddresse.com # so you'll get the output of your cron jobs by mail
*/5 * * * * /usr/local/bin/resubmit_jobs # this add a cron jobs that will run your resubmit_jobs script every 5 minutes

If you want to run this script manually you just have to make: resubmit_jobs

iterations=200
while [[ $iterations -gt 1 ]]; do
    for job in $(alien_ps -E | grep 'E' | awk '{print $2}' | sed 's/-//'); do
        alien_resubmit $job
    done
    sleep 300
    iterations=$(($iterations-1))
done
  • Hi Thanks. This command worked in interactive window. can you please also write the while loop structure for 5min interval? So that I make a 'testjob.sh' executable srcipt that I run once and it keeps running your command for each 5min interval for N (= 200) times. I have never wrote any loop in shell script. – R. Haq Nov 3 '16 at 23:28
  • Hello, @R.Haq. I have added the 5 minute delay and two hundred iterations of the earlier verison. Keep in mind that this means that the script will run for a little over two thirds of a day. – DopeGhoti Nov 4 '16 at 15:33

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.