2

I am attempting to look into multiple subdirectories within a single directory, and plot the files in each subdirectory using a python script that I am calling within this bash script using gnu-parallel.

The arguments the python script takes are -p: path to files (i.e. the subdirectory) and -o: outpath for plots (which I want to be the same as -p).

I have this bash script:

#!/bin/bash

script="/path/to/python/script/plot.py"

files=($(find . -maxdepth 2 -mindepth 2 | sort))

pat=$files
out=$files
filt="True"
chan="Z"

parallel --jobs 4 python $script -f {} -p $pat -o $out -fl $filt -c $chan ::: ${files[@]}

However, there are no plots in each subdirectory, and the script is running really fast, so I assume that nothing is being piped into the pat or out arguments. What am I doing wrong? Thanks!

2
  • if you add set -euo pipefail after the shebang line #!/bin/bash it will give you more info on what your script is doing
    – user86219
    Commented Mar 27, 2020 at 21:00
  • Try: parallel --dry-run .... It will tell you what GNU Parallel is running. Maybe filenames contain space?
    – Ole Tange
    Commented Mar 29, 2020 at 13:55

1 Answer 1

2

I don't think $pat and $out have the values you think they do

$ echo ${files[@]}
./d1/file1 ./d1/file2 ./d1/file3 ./d2/file4 ./d2/file5 ./d2/file6
$ pat=$files
$ echo ${pat[@]}
./d1/file1
$ out=$files
$ echo ${pat[@]}
./d1/file1

I think what you're trying to accomplish requires different values for $pat and $out depending on the value of parallel's internal “variable” ‘{}’ (which would be each member of the $files array), but what you've got only puts the first member of $files on the command line.

Among the other “variables” GNU Parallel supports, one is ‘{//}’, which is replaced by the directory name of the ‘{}’ value (if any).

Given that, I think you want something more like

#!/bin/bash

script="/path/to/python/script/plot.py"

files=($(find . -maxdepth 2 -mindepth 2 | sort))

filt="True"
chan="Z"

parallel --jobs 4 python $script -f {} -p {//} -o {//} -fl $filt -c $chan ::: ${files[@]}

Is this something more like what you want?

$ parallel echo python $script -f {} -p {//} -o {//} -f $filt -c $chan ::: ${files[@]}
python /path/to/python/script/plot.py -f ./d1/file1 -p ./d1 -o ./d1 -f True -c Z
python /path/to/python/script/plot.py -f ./d1/file2 -p ./d1 -o ./d1 -f True -c Z
python /path/to/python/script/plot.py -f ./d1/file3 -p ./d1 -o ./d1 -f True -c Z
python /path/to/python/script/plot.py -f ./d2/file5 -p ./d2 -o ./d2 -f True -c Z
python /path/to/python/script/plot.py -f ./d2/file4 -p ./d2 -o ./d2 -f True -c Z
python /path/to/python/script/plot.py -f ./d2/file6 -p ./d2 -o ./d2 -f True -c Z

Since you're using sort on the find, I think you may also want to use the --keep-order parameter:

$ parallel --keep-order echo python $script -f {} -p {//} -o {//} -f $filt -c $chan ::: ${files[@]}
python /path/to/python/script/plot.py -f ./d1/file1 -p ./d1 -o ./d1 -f True -c Z
python /path/to/python/script/plot.py -f ./d1/file2 -p ./d1 -o ./d1 -f True -c Z
python /path/to/python/script/plot.py -f ./d1/file3 -p ./d1 -o ./d1 -f True -c Z
python /path/to/python/script/plot.py -f ./d2/file4 -p ./d2 -o ./d2 -f True -c Z
python /path/to/python/script/plot.py -f ./d2/file5 -p ./d2 -o ./d2 -f True -c Z
python /path/to/python/script/plot.py -f ./d2/file6 -p ./d2 -o ./d2 -f True -c Z

Which only makes sure the order of the output is the same as the input, not that the jobs are run in that or any deterministic order.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .