0

I run a lot of small computing jobs in remote cluster where job submission is managed by PBS. Normally in a PBS (bash) script I specify the queue that I would like to submit the job through the command

#PBS -q <queue_name>

The job queue that I need to chose depends on the load on a specific queue. Every time before I submit a job, I analyze this through the command on terminal

qstat -q

which provides me an output which looks like as follows

Queue            Memory CPU Time Walltime Node  Run Que Lm  State
---------------- ------ -------- -------- ----  --- --- --  -----
queue1           --      --    03:00:00   --    0   2 --   E R
queue2           --      --    06:00:00   --    8   6 --   E R

I would like to automate the queue selection by the job script based on two constraints

  1. The queue selected must have a walltime more than the job time specified. The job time is specified through command #PBS -l walltime=02:30:00.
  2. The queue must have the least no. of jobs in Que as shown in the above output.

I'm having trouble in identifying which tools that I need to use in terminal to help me automate the queue selection

0

I doubt you will find such tools as what you describe sounds like a broken PBS setup.

Queues should differentiate by intended usage/users. If both queues serve the same users with the same requirements, they should be merged specifically to negate issues such as the one you raised. The entire point of batch schedulers is to juggle resources between users/jobs. If a user has to manually add input into the equation, something in the process is broken in my opinion.
Not a good solution, but you could just write a script on your own to get the queue, something like:

#!/bin/bash
case $# in
  2) ;;
  *) echo "Usage: $(basename $0) 'WALLTIME' 'JOBNAME'"
  echo "WALLTIME should be in format HH:MM:SS"
  echo "JOBNAME should be filename of PBS script (full path if not in local directory)"
  exit 0
  ;;
esac

MYWALL="$1"
MYJOB="$2"
cnt=0
i=0
qstat -q|
while read q
do
   if (( cnt < 2 ))
   then
      ((cnt++))
      continue
   else
      ((i++))
      WALL[$i]=$(echo $q|awk '{print $4}')
      Q[$i]=$(echo $q|awk '{print $7}')
      NAME[$i]=$(echo $q|awk '{print $1}')
   fi
done
MINQ=${Q[1]}
for ((q=1; q <= i; q++)) 
do
   if [[ "${MYWALL}" > "${WALL[$q]}" ]]
   then
      continue
   else
      if [[ ${Q[$q]} -le $MINQ ]]
      then
         MINQ=${Q[$q]}
         NAMEQ="${NAME[$q]}"
      fi
   fi
done

if [[ ! "$NAMEQ" ]]
then
   echo "WALLTIME $MYWALL greater than walltime of any queue"
fi
echo "qsub -q $NAMEQ $MYJOB"
qsub -q "$NAMEQ" "$MYJOB"
  • My apologies. I was actually looking for a script that can automate this process. I will try this script on my queue and accept the answer once I succeed. – WanderingMind Jul 16 '14 at 19:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.