find -exec and increment counter / progress

Question

TL;DR:

Q: How to keep a counter in find -exec loop ?

---

My use-case:

I need to move a lot of directories which are scattered around the place, so I do

find . -type d -name "prefix_*" \
    -exec sh -c '
        new_path="/new/path/$(basedir "$1")";
        [ -d "$new_path" ] || mv "$1" "$new_path";
    ' find_sh {} \;

(The real command is more complex, as I read some metadata for the constitution of /new/path. Anyways, I do not want to argue about the command itself, it's not part of the question, just the use-case).

It works just fine, but it takes quite a while and I want to keep track of the progress.

So I added a counter writing to a file:

i=$(cat ~/find_increment || echo 0);
echo $((i+1)) | tee ~/find_increment;

That also works just fine, but it feels like a really bad idea having some 100.000 disk read and write operations.

I thought about writing to ramdisk instead of disk, but I don't have that option in the environment I need to perform that task.

Is there a better way to keep a counter between -exec runs ?

Answer 1

Instead of using a pure find command you could combine find with a while read loop or GNU parallel. Both are likely to be faster than find's -exec since you don't start a new shell for every path found by find.

Solution Using GNU Parallel

GNU parallel has the following benefits compared to while read:

• Easier to get right. No IFS= and -r needed.
• Built-in job number variable {#}.
  For more handy replacement strings have a look at the tutorial.
• Easy to parallelize if needed.
  Remove the -j1 and you have as many workers as cores by default.

script='
    echo Processing job number {#}
    new_path="/new/path/$(basedir {})"
    [ -d "$new_path" ] || mv {} "$new_path"
'
find … -print0 | parallel -0 -j1 "$script"

The {} is replaced by parallel by the correctly quoted entry read from stdin. Do not quote {} again.

parallel executes the script with the same shell from which you started it. If you started parallel in bash you can use bash features in the script.

Solution Using While Read

find … -print0 |
while IFS= read -r -d '' old_path; do
    echo Processing job number "$((++job))"
    new_path="/new/path/$(basedir "$old_path")"
    [ -d "$new_path" ] || mv "$old_path" "$new_path"
done 

Answer 2

If available, store the counter in /dev/shm/ to prevent disk writes.

=> Use /dev/shm/find_increment instead of ~/find_increment.