My scripts are having trouble with correctly running things in GNU parallel.

I have a sub_script like so (all these are actually simplified versions):

#! /bin/bash   
# input is a date in YYYYMMDD format  
mkdir -p $input

cd $input  
echo 'line1' > $filename  
echo 'The date is: '$input >> $filename

Then I have a file multi.sh like so:

cd /home/me/scripts; ./sub_script 20141001   
cd /home/me/scripts; ./sub_script 20141002   
cd /home/me/scripts; ./sub_script 20141003   
cd /home/me/scripts; ./sub_script 20141004   
cd /home/me/scripts; ./sub_script 20141005    

I am trying to use GNU parallel to execute all these functions with multiple cores using this command

parallel -j 3 --delay 1 < multi.sh 

to run on 3 cores. I've tried to implement a 1 second delay between running each line to prevent problems, but this does not work.

I am having problems with the new directories containing improper files. I think this only happens when there are more lines in multi.sh than cores specified by -j, and it only happens sporadically (it's not always reproducible). I can rerun the parallel line 2 times in a row and get different results. Sometimes I might get 20141002.txt files in the 20141005 directory instead of the 20141005.txt files. Other times I may only get the 20141002.txt files in the 201005 directory.

Are there any suggestions on how I can fix this? GNU parallel is preferred, but I can try other commands as well.

  • 1
    I don't see how it could be causing your issue but as a genera rule, don't do cd foo; ./script, just run foo/script directly. For example /home/me/scripts/sub_script 20141003. Can you actually reproduce the error with the minimal example you show? I'm afraid you might have simplified it too much and the real issue is somewhere else, hidden in the code you run.
    – terdon
    Commented Nov 19, 2014 at 0:58
  • You code looks sound and GNU Parallel should no do that. I cannot reproduce your problem on my system. Can you reproduce the problem on one of these VirtualBox images: sourceforge.net/projects/virtualboximage/files
    – Ole Tange
    Commented Nov 19, 2014 at 7:55
  • @terdon mkdir -p $input creates the directories (line 4)
    – Anthon
    Commented Nov 19, 2014 at 8:39
  • @terdon Ah, good point. I had the cd in there based on a batchfile from before. I failed to notice that it was no longer needed. I have not actually been able to reproduce the error with the minimal example. Additional details on my system/project: RedHat server using ssh to access, got error running 11 days on 8 processors (and other times), the subscript calls in Fortran code (after changing directories) that actually creates the files in the folder where it is run. It may not be GNU Parallel, that was just a guess. Thanks to all!
    – Jake112
    Commented Nov 21, 2014 at 1:07
  • @Anthon whoops, indeed. I missed that. I will now abuse my mod powers and edit my last comment.
    – terdon
    Commented Nov 21, 2014 at 1:38

1 Answer 1


Why the extra batchfile, if you use parallel?

parallel -j3 --delay 1 ./sub_script ::: 20141001 20141002 20141003 20141004 20141005
  • Thanks. I've reran it about a dozen times this way and I have not had the problem again. This seems to have solved it. It's much cleaner than my original approach.
    – Jake112
    Commented Nov 21, 2014 at 0:50

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .