Hi all I have the following bash script that calls hmmscan from hmmer3 software. hmmscan requires to specify 6 command line arguments in this case the code that I have written is as follows:

while getopts i:t:o:m:e: option
    case "${option}"
            i) in_dir=${OPTARG};;
            t) n_threads=${OPTARG};;
            o) out_dir=${OPTARG};;
            m) marker_profiles=${OPTARG};;
            e) extension=${OPTARG};;

rm -rf $out_dir
mkdir $out_dir

for file in $(ls $in_dir*$extension)
        echo $file
        echo $extension
        #fname=$(echo $file | cut -d'/' -f2)
        fname=$(echo $file | rev | cut -d'/' -f1 | rev)
        echo $out_dir$fname".txt"
        hmmscan -o $out_dir$fname".txt" --tblout $out_dir$fname".hmm" -E 1e-10 --cpu $n_threads $marker_profiles  $file

basically I'm iterating over a list of files found in a directory and am running hmmscan over each file, and I'm using this file name to append on the output names so that I'll have different output names corresponding to each of my input files. My question is that the loop is quite length and I would like to parallelize this process to scale with the number of CPUs that I provide at command line. Unfortunately I do not know how to do so where there is many inputs to my command within the for loop. Any help would be appreciated. Many thanks.

UPDATES!! I have converted it and put it in a function as follows:

hmmscan_fun () {
    local file=$1
    local marker_profiles=$2
    local n_threads=$3
    local out_dir=$4
    fname=$(echo $file | rev | cut -d'/' -f1 | rev)
    echo 'filename'
    echo $out_dir$fname".txt"
    echo 'n threads'
    echo $n_threads
    echo 'marker profiles'
    echo $marker_profiles
    echo $out_dir$fname".txt" >> $out_dir"out.txt"

for file in $(ls $in_dir*$extension)
    hmmscan_fun "$file" "$marker_profile" "$n_threads" "$out_dir" &

echo 'Finished HMMSCAN'

However now I have one more small problem the script is not getting out of the loop and executing my last line of code which is

echo 'finished HMMSCAN'

any idea how to fix this?

  • Write the part in the loop as a function and call it with parallel or xargs -P. Or use sem. --> see this. And: Don't parse ls. – pLumo Jul 10 '19 at 7:51
  • thank you for this I have written the code as a function I will update my main post however there's one more issue now the script is not terminating. It is going over all the files and when it reaches the last it is just not stopping, and it's not returning me bash cursor '>>>' How should I end the for loop or the function to do so?> – user3289556 Jul 10 '19 at 14:04
  • It is actually halting but since im executing over so many cores the output was getting messed up I just put a simple pause command towards the end to stop and catch up. Thank you all. – user3289556 Jul 10 '19 at 14:16
  • 1
    You can prevent messing up the output using flock. If you use & you spawn as many processes as possible. To limit that, use parallel. And again: Don't parse ls. – pLumo Jul 10 '19 at 14:17
  • okay so the & is breaking everything and killing the server I cant understand what it's doing. Could you provide me some syntax how to use parallel with functions and a for loop to pass the different arguments? I cannot find anything online – user3289556 Jul 10 '19 at 15:19

Browse other questions tagged or ask your own question.