How can I recursively find directories and parse them into python script call within bash script?

Question

I am attempting to look into multiple subdirectories within a single directory, and plot the files in each subdirectory using a python script that I am calling within this bash script using gnu-parallel.

The arguments the python script takes are -p: path to files (i.e. the subdirectory) and -o: outpath for plots (which I want to be the same as -p).

I have this bash script:

#!/bin/bash

script="/path/to/python/script/plot.py"

files=($(find . -maxdepth 2 -mindepth 2 | sort))

pat=$files
out=$files
filt="True"
chan="Z"

parallel --jobs 4 python $script -f {} -p $pat -o $out -fl $filt -c $chan ::: ${files[@]}

However, there are no plots in each subdirectory, and the script is running really fast, so I assume that nothing is being piped into the pat or out arguments. What am I doing wrong? Thanks!

Answer 1

I don't think $pat and $out have the values you think they do

$ echo ${files[@]}
./d1/file1 ./d1/file2 ./d1/file3 ./d2/file4 ./d2/file5 ./d2/file6
$ pat=$files
$ echo ${pat[@]}
./d1/file1
$ out=$files
$ echo ${pat[@]}
./d1/file1

I think what you're trying to accomplish requires different values for $pat and $out depending on the value of parallel's internal “variable” ‘{}’ (which would be each member of the $files array), but what you've got only puts the first member of $files on the command line.

Among the other “variables” GNU Parallel supports, one is ‘{//}’, which is replaced by the directory name of the ‘{}’ value (if any).

Given that, I think you want something more like

#!/bin/bash

script="/path/to/python/script/plot.py"

files=($(find . -maxdepth 2 -mindepth 2 | sort))

filt="True"
chan="Z"

parallel --jobs 4 python $script -f {} -p {//} -o {//} -fl $filt -c $chan ::: ${files[@]}

Is this something more like what you want?

$ parallel echo python $script -f {} -p {//} -o {//} -f $filt -c $chan ::: ${files[@]}
python /path/to/python/script/plot.py -f ./d1/file1 -p ./d1 -o ./d1 -f True -c Z
python /path/to/python/script/plot.py -f ./d1/file2 -p ./d1 -o ./d1 -f True -c Z
python /path/to/python/script/plot.py -f ./d1/file3 -p ./d1 -o ./d1 -f True -c Z
python /path/to/python/script/plot.py -f ./d2/file5 -p ./d2 -o ./d2 -f True -c Z
python /path/to/python/script/plot.py -f ./d2/file4 -p ./d2 -o ./d2 -f True -c Z
python /path/to/python/script/plot.py -f ./d2/file6 -p ./d2 -o ./d2 -f True -c Z

Since you're using sort on the find, I think you may also want to use the --keep-order parameter:

$ parallel --keep-order echo python $script -f {} -p {//} -o {//} -f $filt -c $chan ::: ${files[@]}
python /path/to/python/script/plot.py -f ./d1/file1 -p ./d1 -o ./d1 -f True -c Z
python /path/to/python/script/plot.py -f ./d1/file2 -p ./d1 -o ./d1 -f True -c Z
python /path/to/python/script/plot.py -f ./d1/file3 -p ./d1 -o ./d1 -f True -c Z
python /path/to/python/script/plot.py -f ./d2/file4 -p ./d2 -o ./d2 -f True -c Z
python /path/to/python/script/plot.py -f ./d2/file5 -p ./d2 -o ./d2 -f True -c Z
python /path/to/python/script/plot.py -f ./d2/file6 -p ./d2 -o ./d2 -f True -c Z

Which only makes sure the order of the output is the same as the input, not that the jobs are run in that or any deterministic order.