0

I am using this script to analyse data.

#!/bin/bash
inlist=/aut/cl/tera/mja/scripts
in=/aut/cl/tera/mja/data/ran_ready/
out=/aut/cl/tera/mja/data/mas/aft/
for i in $(cat $inlist/list.txt); do
echo "$i" 
mri_binarize --i ${in}/${i}/mri/aseg.mgz --o ${out}/${i}/masks/cc.nii.gz --match 41 42 43 44 45 
flirt -in ${out}/${i}/masks/cc.nii.gz -ref ${out}/${i}/T1.nii -out ${out}/${i}/masks/cc2T1.nii.gz -omat ${out}/${i}/masks/cc2T1.mat -bins 256 -cost corratio -searchrx -90 90 -searchry -90 90 -searchrz -90 90 -dof 12  -interp trilinear

echo "$i ... done"    
wait
done

This script is running two command lines on a list of entries (50 entries ) mentioned in list.txt. For every entry this script needs half an hour to finish the job. I wanted to run this script for all the entries in list.txt at the same time using "xargs" or any other possible way!

How can we revise this script to make it more efficient?

1

If the programs don't have any problem running in parallel, why not simply send them to the background:

for i in $(cat $inlist/list.txt); do    
    echo "$i" 
    ( 
        mri_binarize --i ${in}/${i}/mri/aseg.mgz --o ${out}/${i}/masks/cc.nii.gz --match 41 42 43 44 45 
        flirt -in ${out}/${i}/masks/cc.nii.gz -ref ${out}/${i}/T1.nii -out ${out}/${i}/masks/cc2T1.nii.gz -omat ${out}/${i}/masks/cc2T1.mat -bins 256 -cost corratio -searchrx -90 90 -searchry -90 90 -searchrz -90 90 -dof 12  -interp trilinear
        echo "$i ... done"        
    ) &
done

The parentheses group the commands together and run them in a subshell, so we can send the commands together to the background. I am assuming that the flirt command needs to run after mri_binarize. If not, simply tack on an & after both commands:

mri_binarize --i ${in}/${i}/mri/aseg.mgz --o ${out}/${i}/masks/cc.nii.gz --match 41 42 43 44 45 &
flirt -in ${out}/${i}/masks/cc.nii.gz -ref ${out}/${i}/T1.nii -out ${out}/${i}/masks/cc2T1.nii.gz -omat ${out}/${i}/masks/cc2T1.mat -bins 256 -cost corratio -searchrx -90 90 -searchry -90 90 -searchrz -90 90 -dof 12  -interp trilinear &
| improve this answer | |
  • Yes flirt command needs to run after mri-binarize. In this way can I add more command after flirt? and what is the purpose of the brakets? – user88036 Oct 27 '14 at 19:03
  • @MJA yes, using the first method, add more commands before the closing parenthesis ). – muru Oct 27 '14 at 19:04
  • 1
    @MJA the brackets group the commands, so we can send a set of commands together into the background. – muru Oct 27 '14 at 19:07
  • Sorry. It is still running one by one! – user88036 Oct 27 '14 at 19:12
  • @MJA the is being echoed don't come up together? Did you remove the wait in the for loop? – muru Oct 27 '14 at 19:13
3

If you have GNU Parallel you can do this:

doit() {
  i=$1
  echo "$i" 
  mri_binarize --i ${in}/${i}/mri/aseg.mgz --o ${out}/${i}/masks/cc.nii.gz --match 41 42 43 44 45 
  flirt -in ${out}/${i}/masks/cc.nii.gz -ref ${out}/${i}/T1.nii -out ${out}/${i}/masks/cc2T1.nii.gz -omat ${out}/${i}/masks/cc2T1.mat -bins 256 -cost corratio -searchrx -90 90 -searchry -90 90 -searchrz -90 90 -dof 12  -interp trilinear
  echo "$i ... done"
}
export -f doit

parallel doit :::: $inlist/list.txt

This will run 1 job per core.

All new computers have multiple cores, but most programs are serial in nature and will therefore not use the multiple cores. However, many tasks are extremely parallelizeable:

  • Run the same program on many files
  • Run the same program for every line in a file
  • Run the same program for every block in a file

GNU Parallel is a general parallelizer and makes is easy to run jobs in parallel on the same machine or on multiple machines you have ssh access to.

If you have 32 different jobs you want to run on 4 CPUs, a straight forward way to parallelize is to run 8 jobs on each CPU:

Simple scheduling

GNU Parallel instead spawns a new process when one finishes - keeping the CPUs active and thus saving time:

GNU Parallel scheduling

Installation

If GNU Parallel is not packaged for your distribution, you can do a personal installation, which does not require root access. It can be done in 10 seconds by doing this:

(wget -O - pi.dk/3 || curl pi.dk/3/ || fetch -o - http://pi.dk/3) | bash

For other installation options see http://git.savannah.gnu.org/cgit/parallel.git/tree/README

Learn more

See more examples: http://www.gnu.org/software/parallel/man.html

Watch the intro videos: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Walk through the tutorial: http://www.gnu.org/software/parallel/parallel_tutorial.html

Sign up for the email list to get support: https://lists.gnu.org/mailman/listinfo/parallel

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy