0

I have a large script which relies on input arguments (with getopts). One of these arguments is a directory containing files (all named *bam) This script has 2 parts:

  • Part1: based on input *bam files, calculate one specific number. To be clear, the result is one single number, NOT one number per file.
  • Part 2: using the number found in part1, perform a series of operations on each *bam file.

Now, originally, part1 was very quick, computationally speaking. So my setup was:

  • Run script on terminal: bash script.sh
  • Within the script.sh, for part2, make a HPC job submission for each file

However, now that I need to analyze many more files than originally planned, I am realising that Part1 will also be computationally heavy - I therefore need to also run this on the HPC.

So my question is:

  • Is it possible to submit an HPC job which submits jobs in it?
  • In other words, can I submit script.sh as a job and and still have it submit jobs in its part2?

To be clear, here is an example of what my script might look like:

#!/usr/bin/bash

# PART 0: accept all input arguments

USAGE() { echo "Usage: bash $0 [-b <in-bam-files-dir>] [-o <out-dir>] [-c <chromlen>]" 1>&2; exit 1; }

if (($# == 0)); then
        USAGE
fi

# Use getopts to accept each argument

while getopts ":b:o:c:h" opt
do
    case $opt in
       b ) BAMFILES=$OPTARG
        ;;
       o ) OUTDIR=$OPTARG
        ;;
       c ) CHROMLEN=$OPTARG
        ;;
       h ) USAGE
        ;;
       \? ) echo "Invalid option: -$OPTARG exiting" >&2
        exit
        ;;
       : ) echo "Option -$OPTARG requires an argument" >&2
        exit
        ;;
        esac
    done

# PART1: calculate this unique number

NUMBER=0    

for i in  $(ls $BAMFILES/*.bam)
do
  make some calculations on each file to obtain a number ...
  keep only the smallest found number and assign its value to $NUMBER
done

echo "Final number is ${NUMBER} "

# PART2: Using $NUMBER that we found above, submit a job for each *bam file

for i in $(ls $BAMFILES/*bam)
do

    if [ ! -f ${OUTDIR}/${SAMPLE}.bw ];
    then 
        command=" command -options -b $NUMBER $i"

    echo $command | qsub -V -cwd -o $OUTDIR -e $OUTDIR -l tmem=6G -l h_vmem=6G -l h_rt=3600 -N result_${SAMPLE}

    fi

done

1 Answer 1

0

The answer is "it depends". Your HPC cluster can be set up to have execute node able to submit jobs but this isn't a requirement. Sounds like a quick question to your local HPC admin will give you a definitive answer. Or you could try a quick script that does nothing but submit a second job and see if it works.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .