2

My scripts are having trouble with correctly running things in GNU parallel.

I have a sub_script like so (all these are actually simplified versions):

#! /bin/bash   
input=$1  
# input is a date in YYYYMMDD format  
mkdir -p $input

cd $input  
filename=$input'.txt'  
echo 'line1' > $filename  
echo 'The date is: '$input >> $filename

Then I have a file multi.sh like so:

cd /home/me/scripts; ./sub_script 20141001   
cd /home/me/scripts; ./sub_script 20141002   
cd /home/me/scripts; ./sub_script 20141003   
cd /home/me/scripts; ./sub_script 20141004   
cd /home/me/scripts; ./sub_script 20141005    

I am trying to use GNU parallel to execute all these functions with multiple cores using this command

parallel -j 3 --delay 1 < multi.sh 

to run on 3 cores. I've tried to implement a 1 second delay between running each line to prevent problems, but this does not work.

I am having problems with the new directories containing improper files. I think this only happens when there are more lines in multi.sh than cores specified by -j, and it only happens sporadically (it's not always reproducible). I can rerun the parallel line 2 times in a row and get different results. Sometimes I might get 20141002.txt files in the 20141005 directory instead of the 20141005.txt files. Other times I may only get the 20141002.txt files in the 201005 directory.

Are there any suggestions on how I can fix this? GNU parallel is preferred, but I can try other commands as well.

  • 1
    I don't see how it could be causing your issue but as a genera rule, don't do cd foo; ./script, just run foo/script directly. For example /home/me/scripts/sub_script 20141003. Can you actually reproduce the error with the minimal example you show? I'm afraid you might have simplified it too much and the real issue is somewhere else, hidden in the code you run. – terdon Nov 19 '14 at 0:58
  • You code looks sound and GNU Parallel should no do that. I cannot reproduce your problem on my system. Can you reproduce the problem on one of these VirtualBox images: sourceforge.net/projects/virtualboximage/files – Ole Tange Nov 19 '14 at 7:55
  • @terdon mkdir -p $input creates the directories (line 4) – Anthon Nov 19 '14 at 8:39
  • @terdon Ah, good point. I had the cd in there based on a batchfile from before. I failed to notice that it was no longer needed. I have not actually been able to reproduce the error with the minimal example. Additional details on my system/project: RedHat server using ssh to access, got error running 11 days on 8 processors (and other times), the subscript calls in Fortran code (after changing directories) that actually creates the files in the folder where it is run. It may not be GNU Parallel, that was just a guess. Thanks to all! – Jake112 Nov 21 '14 at 1:07
  • @Anthon whoops, indeed. I missed that. I will now abuse my mod powers and edit my last comment. – terdon Nov 21 '14 at 1:38
1

Why the extra batchfile, if you use parallel?

parallel -j3 --delay 1 ./sub_script ::: 20141001 20141002 20141003 20141004 20141005
  • Thanks. I've reran it about a dozen times this way and I have not had the problem again. This seems to have solved it. It's much cleaner than my original approach. – Jake112 Nov 21 '14 at 0:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.