Bash: join pipes without deadlock

Question

I want to list a bunch of filenames via find, pipe them through a utility (let's call this util) which outputs a new name for each input name, and then rename each file from the old name to the new.

The most basic solution would be this:

find . -print0 | while IFS= read -d '' -r old_name; do
    new_name="$(echo "$file" | util)"
    mv "$old_name" "$new_name"
done

The problem with this approach is that util is too slow to fire up for each filename separately. So the solution is to launch util only once and pipe all the filenames through this single process:

find . -print0 >old_names
util <old_names >new_names

exec {old_fd}<old_names
exec {new_fd}<new_names

while IFS= read -d '' -r old_name <&$old_fd &&
      IFS= read -d '' -r new_name <&$new_fd; do
    mv "$old_name" "$new_name"
done

This will launch util only once, on the other hand it's no longer a pipeline: we have to list all the files into a tmp file, run util on this tmp file to get another tmp file, and only then do we actually start the renaming...

I've tried the following to do it in a pipelined way:

mkfifo old_names new_names
find . -print0 | tee old_names | util >new_names &

exec {old_fd}<old_names
exec {new_fd}<new_names

while IFS= read -d '' -r old_name <&$old_fd &&
      IFS= read -d '' -r new_name <&$new_fd; do
    mv "$old_name" "$new_name"
done

Unfortunately this can deadlock depending on how util does input/output buffering...

So my question is: what's the proper way of doing this in bash?

Answer 1

A solution without joining pipes:

find . -print > infiles
cat infiles | util > outfiles
parallel mv ::::+ infiles outfiles

Pro: Extremely simple. Con: renaming only starts after util is done. 2 tempfiles.

find . -print > infiles
cat infiles | util | parallel -j1 mv ::::+ infiles -

Pro: Renaming will start when util starts giving names. Con: 1 tempfile.

find . -print | util | parallel -j1 mv ::::+ <(find . -print) -

Pro: Renaming will start when util starts giving names. Con: requires no change to the current directory (otherwise the two finds may not give the same.)

Answer 2

Depending on the complexity of the transform (util), you may be able to use rename (the one by Larry Wall).

Alternatively

Arrange util to output name pairs: old-name new-name, then

find . -print0 | util | xargs -n2 mv -T

Alternatively

Try something like this.

function a() { 
    echo -n "$1 "
    echo "$1" | sed -e s/a/b/g
}

… | while read arg; do
a "$arg"
done