0

I would like to parallelise the for loop in the bash script below.


I have two Matlab scripts

1) main.m

clear
rng default
P=2;
grid=randn(2,3);
jobs=1;

2) f.m

sgetasknum_grid=grid(jobs*(str2double(getenv('SGE_TASK_ID'))-1)+1: str2double(getenv('SGE_TASK_ID'))*jobs,:); %jobsx3

result=sgetasknum_grid+1; 

filename = sprintf('result.%d.mat', ID);
save(filename, 'result')

exit

What I want to do:

  • I want to run main.m;

  • then, I want to run f.m twice in parallel

  • Everything should be executed on node A


Here's my version without parallelising

1) I save main.m and f.m into a folder named My_folder

2) I create the bash file td.sh as below and save it into the folder My_folder

    #$ -S /bin/bash
    #$ -l h_vmem=5G
    #$ -l tmem=5G
    #$ -l h_rt=480:0:0
    #$ -cwd
    #$ -j y


    #$ -N try

    date
    hostname

    export SGE_TASK_ID

    for SGE_TASK_ID in {1..2}
do
    #Output the Task ID
    echo "Task ID is $SGE_TASK_ID"

  /share/.../matlab -nodisplay -nodesktop -nojvm -nosplash -r "main; ID=$SGE_TASK_ID; f; exit"
done

3) I go into the terminal and type ssh username@A, then cd /.../My_folder, then bash td.sh


In order to parallelise, many answers in this forum suggests to use parallel. I guess the bash file would look something like the following

#$ -S /bin/bash
#$ -l h_vmem=5G
#$ -l tmem=5G
#$ -l h_rt=480:0:0
#$ -cwd
#$ -j y


#$ -N try

date
hostname

export SGE_TASK_ID

SGE_TASK_ID={1..2}

echo "$SGE_TASK_ID" | parallel -P 2 /share/.../matlab -nodisplay -nodesktop -nojvm -nosplash -r "main; ID=$SGE_TASK_ID; f; exit"

However, this is the error that I get on the terminal

A.local
td.sh: line 16: parallel: command not found

I understand that it may be that the parallel "software" is missing in the machine. Can I install it and if yes how? Or should it be installed by the sysadmins? Alternatively, do you have other recommendations instead of parallel?


Update: script that attempts to implement the second answer below

#!/bin/bash -l
#$ -S /bin/bash
#$ -l h_vmem=5G
#$ -l tmem=5G
#$ -l h_rt=480:0:0
#$ -cwd
#$ -j y


#$ -N try

date
hostname

export SGE_TASK_ID

SGE_TASK_ID={1..2}

echo -e SGE_TASK_ID | xargs -I {} -P 4 /share/.../matlab -nodisplay -nodesktop -nojvm -nosplash  -r "main; ID={}; f; exit"

And this is the error message that I get

Undefined function or variable 'SGE_TASK_ID'.
2

In your loop, start the Matlab jobs in the background, then wait for them to finish after the loop:

for SGE_TASK_ID in 1 2; do
    printf 'Task ID is %d\n' "$SGE_TASK_ID"

    /share/.../matlab -nodisplay -nodesktop -nojvm -nosplash \
        -r "main; ID=$SGE_TASK_ID; f; exit" &
done

wait

The Matlab jobs will be started in quick succession and will run concurrently. The wait after the loop ensures that the script does not exit until all background jobs are finished.

For J jobs where you are allowed to run N jobs in parallel:

SGE_TASK_ID=1
n=0
while [ "$SGE_TASK_ID" -le "$J" ]; do
    if [ "$n" -eq "$N" ]; then
        wait  # waits for the N started jobs to finish
        n=0
    fi

    printf 'Task ID is %d\n' "$SGE_TASK_ID"

    /share/.../matlab -nodisplay -nodesktop -nojvm -nosplash \
        -r "main; ID=$SGE_TASK_ID; f; exit" &

    SGE_TASK_ID=$(( SGE_TASK_ID + 1 ))
    n=$(( n + 1 ))
done

wait

Another variation is to wait for a single background job to finish inside the loop's if-statement, with wait -n and just decrease n by one instead of setting it to zero.

  • 1
    @user3285148 See update answer. I would assume that the resource limits would be for the resources needed to run N jobs at once. – Kusalananda Dec 7 '18 at 10:58
  • 1
    @user3285148 1) Yes, wait waits for all background jobs to finish. Just change wait in the loop to wait -n and n=0 to n=$(( n - 1 )) to wait for a single job to finish. Leave wait after the loop as it is. Wating for a single job to finish may be more efficient use of resources. 2) parallel (and xargs) are typically used to start or feed multiple background jobs with data from a file or command. You could use them here too, but I don't really see a reason for it. 3) Somewhere above the loop. – Kusalananda Dec 7 '18 at 11:10
  • 1
    @user3285148 No need to echo it, but if you do, put the echo on a new line. – Kusalananda Dec 7 '18 at 11:21
  • 1
    @user3285148 That would indicate that the script is not being run by bash. – Kusalananda Dec 7 '18 at 11:34
  • 1
    @user3285148 It means that whatever shell the queuing system uses to run your job submission script is not bash, or at least not a version of bash that knows what wait -n means. You could try adding a proper #!/bin/bash-line at the very top of the script (1st line, no space before the #!), and if that does not work, read the manual for the submission command that you use, or contact a system admin for more info. – Kusalananda Dec 7 '18 at 11:42
1

a slightly cleaner solution;

echo -e "1\n2" | xargs -I {} -P $(nproc) /share/.../matlab -nodisplay -nodesktop -nojvm -nosplash  -r "main; ID={}; f; exit"

if you have more than 2 tasks

  • Thanks. Suppose that I'm allowed to perform at most N tasks in parallel, but the number of iterations of my for loop is J>N. Does your code allow to execute N tasks in parallel and, as soon as one is finished, to "refill" it with a new task until the J tasks are all executed? The other answer gave me that option too. – user3285148 Dec 7 '18 at 13:17
  • Thanks. (1) What are N and J in your code? (2) What do you mean by "a tool that comes with your OS" (sorry, I'm a very beginner, I don't know this terminology)? – user3285148 Dec 8 '18 at 16:57
  • Thanks. Where is J (number of iterations of the four loop)? – user3285148 Dec 10 '18 at 9:45
  • Not sure I'm implementing your code in the correct way. See the updated part of my answer – user3285148 Dec 10 '18 at 10:09
  • Thanks. But SGE_TASK_ID is in my f.m script... – user3285148 Dec 10 '18 at 16:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.