4

I am trying to speed up a find command which accesses files on multiple harddrives by utilizing parallelization. Unfortunately, either the parallelization is ignored or the variable is not filled.

found=""; IFS=$'\n'

for hdd in "${hdd_list[@]}"
do
    found+=$'\n'$(find "$hdd" -name "*filter*" -type f &) # ignores parellelization
    found+=$'\n'$(find "$hdd" -name "*filter*" -type f) & # doesn't fill variable
done

Is this possible at all without resorting to temp files?

4 Answers 4

4

Command substitutions wait to get all the data from the pipe opened to the command inside before they continue. Having the shell continue reading the data in the background while continuing with the script would be far more complicated than is necessary. (Incidentally, this also means that even if the command inside the command substitution does something weird, like forking and exiting the parent first, you still get all the output when the pipe eventually closes (if it does).) So backgrounding a process inside the command substitution ($(... &) doesn't do anything useful.

Putting the whole assingment in a background (foo=... &) also doesn't work, since background jobs necessarily run in a separate process, and there's no method for that background process to change shell variables in the memory of the main shell process.

You could arrange to connect all the find processes to pipes individually, letting them run and print in parallel, but the pipe buffers are only so large, meaning you'd need to read from all of them simultanously too. And that's hard to do in the shell, since you don't have select(). (Well, could be that some shell does have it.)

But all that's far too complicated since the simple solution of using temporary files is right there.

If your find implementation is nice in that it only writes full lines (entries), and you don't care about the order, you could just redirect all the output to a single file:

f=$(mktemp)
for hdd in "${hdd_list[@]}"; do
    find "$hdd" ... &
done >> "$f"
# read "$f"
rm -f "$f"

But if that's not the case, or you want to be sure, make a temporary directory and one output file for each find:

d=$(mktemp -d)
i=1
for hdd in "${hdd_list[@]}"; do
    find "$hdd" > "$d/out$i.tmp" &
    i=$((i+1))
done
cat "$d"/*.tmp > "$d/all.out"
# read "$d/all.out"
rm -rf "$d"

Though of course in that first one, you could just skip the temporary file and read from the loop directly.

Since you're using a shell with arrays, you might want to put the output from find into an array too. E.g. with readarray in Bash, this would put each line in a distinct array element:

readarray -t files < <(find ...)
3

parset is built for this:

parset hd find {} -name \""*filter*"\" -type f ::: "${hdd_list[@]}"

It does, however, use tempfiles. They are cleaned up for you, so you do not have to deal with them.

See more: https://www.gnu.org/software/parallel/parset.html

2

It seems maybe that you don't understand the full implications of using "&".

The nature of the beast for background processes is that you CANNOT access background child process content ... from a parent process (regardless of variables being declared global or not).

The problem you have is the same as outlined in this other posting. Essentially, temp files is the only way to circumvent the issue.

1

Specifically for parallelizing find, check out fdfind which is an improved version of find that is multi-threaded by default.

If you can't use fdfind, try using xargs, which runs commands in parallel. Borrowing from this question, you could try something like this (though I'm not 100% on it so you may need to tinker before it'll work):

found="$( printf "%s\0" "${hdd_list[@]}" | xargs -0 -I {} find {} -name "*filter*" -type f )"

How it works:

printf "%s\0" "${hdd_list[@]}": prints out the list with null characters as delimiters

| xargs -0 -I {} ...: takes stdin with null-delimited inputs (-0) and calls the command that follows. It will find all instances of {} and replace it with the input field.

1
  • You also have to parse the result from find correctly. Currently, you get all the output of all find processes as a single string. Ideally, you'd get an array where each element is a pathname (and where each pathname is allowed to contain spaces etc.).
    – Kusalananda
    Commented Jan 22, 2023 at 23:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .