0

I have a large script which relies on input arguments (with getopts). One of these arguments is a directory containing files (all named *bam) This script has 2 parts:

  • Part1: based on input *bam files, calculate one specific number. To be clear, the result is one single number, NOT one number per file.
  • Part 2: using the number found in part1, perform a series of operations on each *bam file.

Now, originally, part1 was very quick, computationally speaking. So my setup was:

  • Run script on terminal: bash script.sh
  • Within the script.sh, for part2, make a HPC job submission for each file

However, now that I need to analyze many more files than originally planned, I am realising that Part1 will also be computationally heavy - I therefore need to also run this on the HPC.

So my question is:

  • Is it possible to submit an HPC job which submits jobs in it?
  • In other words, can I submit script.sh as a job and and still have it submit jobs in its part2?

To be clear, here is an example of what my script might look like:

#!/usr/bin/bash

# PART 0: accept all input arguments

USAGE() { echo "Usage: bash $0 [-b <in-bam-files-dir>] [-o <out-dir>] [-c <chromlen>]" 1>&2; exit 1; }

if (($# == 0)); then
        USAGE
fi

# Use getopts to accept each argument

while getopts ":b:o:c:h" opt
do
    case $opt in
       b ) BAMFILES=$OPTARG
        ;;
       o ) OUTDIR=$OPTARG
        ;;
       c ) CHROMLEN=$OPTARG
        ;;
       h ) USAGE
        ;;
       \? ) echo "Invalid option: -$OPTARG exiting" >&2
        exit
        ;;
       : ) echo "Option -$OPTARG requires an argument" >&2
        exit
        ;;
        esac
    done

# PART1: calculate this unique number

NUMBER=0    

for i in  $(ls $BAMFILES/*.bam)
do
  make some calculations on each file to obtain a number ...
  keep only the smallest found number and assign its value to $NUMBER
done

echo "Final number is ${NUMBER} "

# PART2: Using $NUMBER that we found above, submit a job for each *bam file

for i in $(ls $BAMFILES/*bam)
do

    if [ ! -f ${OUTDIR}/${SAMPLE}.bw ];
    then 
        command=" command -options -b $NUMBER $i"

    echo $command | qsub -V -cwd -o $OUTDIR -e $OUTDIR -l tmem=6G -l h_vmem=6G -l h_rt=3600 -N result_${SAMPLE}

    fi

done
0

The answer is "it depends". Your HPC cluster can be set up to have execute node able to submit jobs but this isn't a requirement. Sounds like a quick question to your local HPC admin will give you a definitive answer. Or you could try a quick script that does nothing but submit a second job and see if it works.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.