2

I have a directory which is filled with subdirectories and in each subdirectory 74 files. I have bash script which analyzes the 74 files and wish to run it in parallel on all directories.

When I go into each subdirectory and manually run the bash script the output is perfect.

When attempting to run the bash script in parallel the bash script seems to only be run on one file.

Here is the code I am using to run the script with parallel

parallel "cd {} && bash Final_Fantasy.sh" :::: dir.lst

The strange thing is I have another script that runs on the same files, using the same code as above, and results in 0 issues.

The script I am attempting to run is just somewhat simple writing and overwriting using awk and sed with a few read while loops.

Is something wrong with my parallel code or where should I look for issues?

Thanks!

Edit 1:

1) Script1.sh: Copy Final_Fantasy.sh and a single output file (Output1.txt) into an empty directory.

2) Each directory found in dir.lst now contains only 2 files, Output1.txt and Final_Fantasy.sh

3) The first line of Final_Fantasy.sh cuts Output1.txt into 74 separate files.

4) Final_Fantasy includes multiple lines of awk, sed, and grep to edit one of the 74 files at a time all within a read-while loops.

5) For some of the steps I write the output to a .txt file for later use.

6) The final steps of the loop is to take two .txt files and cat them together.

I also have an exit at the end the file since I was hoping to troubleshoot the loop if anything went wrong. It looks as follows:

done < $filename2 done < $filename1 exit more code that works

The only thing I can think of at this point is 1) something beyond my control with the HPC. 2) exit behaving funny and exiting after the first iteration of the loop instead of exiting after all files have been analyzed.

Edit 2) I am unable to reproduce this issue on my local machine, parallel and all, which leads me to believe it is a strange HPC issue that I don't think I have control over.

Edit 3) Is there a way I can have parallel output what it is doing step-by-step so I can see where the issue is occurring?

  • Your command looks correct, and works for me, so something else is causing this. Can you make an MCVE that works on other systems, too? – Ole Tange Oct 10 '18 at 9:02
  • What is the output of parallel --version? – Ole Tange Oct 10 '18 at 11:08
  • The output is: 20180122 assuming that is the information you wanted. The command works for me as well with other scripts but something about this particular script causes issues. I will update the question to the best of my ability. Thanks for your help – AMB Oct 10 '18 at 20:10
  • You have described the script, but can you make an MCVE? stackoverflow.com/help/mcve – Ole Tange Oct 11 '18 at 0:01
  • For debugging with parallel try: --dry-run, -v, -Dall. – Ole Tange Oct 11 '18 at 0:02
0

parallel executes its arguments directly. You can't do shell stuff unless you use a shell. Put sh -c right after parallel.

Update: The above is only true for moreutils parallel, not GNU parallel, which this question is about.

  • The man page pretty clearly says it can take an executable, or a script. There's even examples obviously running shell snippets. – ilkkachu Oct 10 '18 at 21:54
  • @ilkkachu whoops, I didn't realize there were two different parallel commands and read the man page for the moreutils one instead of the GNU one. Editing the answer now. – Joseph Sible Oct 10 '18 at 23:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.