I have a question regarding linux parallel scripts:

Context: I need to process many .dat files

Code I have so far:

#PBS -S /bin/bash
# set parallel parameters 
#PBS -r n 
#PBS -l walltime=2:00:00 
#PBS -l procs=298 
#PBS -l pmem=1600m 
#PBS -m bea 
#PBS -M myemail@somewhere.com
#PBS -N FileName

cd /home/user/Data/
# start matlab
module load matlab/2015b
# Choose the MCR directory according to the compiler version used
# set up matlab compiler
mcc -R -nodisplay -R -singleCompThread -R -logfile,ResultsOutput.txt -K -m -v -w enable FileName.m

# cycle through all files
# does the for loop send it to different nodes?
for fileID in *.dat
  echo "Running on host: `hostname`"
  echo "Current working directory is `pwd`"
  echo "Starting run at: `date`" 
  echo "$fileID"

  ./run_FileName.sh $MCR $fileID > Results_${PBS_JOBID}.out &

  echo "Job finished at: `date`"    

need to retain results ??

closed as unclear what you're asking by DopeGhoti, Jeff Schaller, Kusalananda, sam, mdpc Jan 27 '17 at 0:09

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • I don't know. Do you need to retain results? – DopeGhoti Jan 26 '17 at 21:38
  • Sorry, but we don't know what type of system you are on. It looks like you're using a queuing system on some sort of compute farm? PBS? – Kusalananda Jan 26 '17 at 21:41
  • What is the question? – G-Man Jan 29 '17 at 0:35

I think you should look at GNU Parallel https://www.gnu.org/software/parallel/.

parallel "./run_FileName.sh $MCR {} > Results_${PBS_JOBID}.out" ::: *.dat

And you probably want to use the option --joblog and --sshloginfile too.

Not the answer you're looking for? Browse other questions tagged or ask your own question.