I have a need to collect some duplicate files and want to avoid name collisions. The problem is that this collection of files might be added to by another execution of my script before the files are cleaned out and just want to continue to increment the number.

I decided on a simple until loop to increment the number like so.

until [[ ! -f ${PWD}/DUPES/${num}-$1 ]]; do
num=$((num +1))
mv --no-clobber $1 ${PWD}/DUPES/${num}-$1

I can't imagine ever exceeding 1k files in an extreme example, so my questions are...

Is this a terribly inefficient way to achieve this? Should I be parsing the existing files to get the highest leading number to start incrementing from there, or is this ok for the extreme example I put forth? Or is there a better way all together?

  • Consider two instances of this running simultaneously, having started simultaneously. They will both find that (say) num=4 is available and both will attempt to use the same target name. – roaima Dec 16 '15 at 20:39
  • If you didn't need sequential numbers, just non-collision, you might be able to use mktemp, possibly with your filename as the base. (We will then assume that mktemp does its job efficiently.) – Ulrich Schwarz Dec 16 '15 at 21:06
  • Perhaps you want a version control system like git – Basile Starynkevitch Dec 17 '15 at 9:46

You might do it like:

set -C; num=0                                  ### set -o noclobber; init $num
[ -e "$1" ] &&                                 ### this needs to be true
until 2>&3 >"./DUPES/$((num+=1))-$1" &&        ### try to open() num+=1
      mv --  "$1"  "./DUPES/$num-$1"           ### if opened, mv over it
do :; done 3>/dev/null                         ### do nothing

You would at once assure that multiple instances cannot secure the same name for any given file, and increment your variable.

The /dev/null < stderr just drops the shell's complaint about a file existing when it tries to do the output truncate/redirect and finds an existing target. While noclobber is enabled it won't overwrite another file - it will only open() a new one unless you use >|. And so you don't need its complaint because the whole point is to increment over existing files until a non-existing name is found.

Regarding the performance aspect - it would be better if you didn't start at zero. Or, if you tried to make up the difference. I guess the above might be improved somewhat like:

set -C; num=0                                  ### set -o noclobber; init $num
until 2>&3 >"./DUPES/$((num+=1))-$1"  &&       ### try to open() num+=1
      mv --  "$1"  "./DUPES/$num-$1"           ### if opened, mv over it
do    [ -e  "./DUPES/$((num*2))-$1" ] &&       ### halve fail distance
      num=$((num*2))                           ### up to a point of course
done  3>/dev/null                              ### done

...but up to 1000 you probably don't have to worry about it terribly. I've got to 65k over random names in a couple seconds.

By the way - you might think you could just:

>"./DUPES/$((num+=1))-$1" mv -- "$1" "./DUPES/$num-$1"

...but it doesn't work in a bash shell.

num=0; echo >"/tmp/$((num+=1))" >&2 "$num"; echo "$num" /tmp/[01]

1 /tmp/1

For whatever reason bash does the assignment in some other context for redirections - and so the expansions happen in a strange order. So you need a separate simple command to expand the correct $num value as I get here with &&. Otherwise, though:

num=0; echo "$((num+=1))" "$num"

1 1
  • I don't think I have to worry about multiple instances, but I never thought of allowing for that either. Being a beginner it took me a few to understand this and I learned a lot just trying to figure it out. As no one mentioned any performance issues for just incrementing till a write can be made vs parsing out where to start incrementing, I'll assume that question is answered as well. Thanks for editing with a clearer explanation as well. – akovia Dec 16 '15 at 21:11
  • @akovia - thanks for the thanks! and sorry i didn't speak to the performance bit in the first place - it's a good question. – mikeserv Dec 16 '15 at 21:40

GNU mv has a --backup option which may be useful. The following interaction shows hoe the files are getting renames when the target exists:

$ touch a b c o
$ mv --backup=numbered --verbose a o
`a' -> `o' (backup: `o.~1~')
$ mv --backup=numbered --verbose b o
`b' -> `o' (backup: `o.~2~')
$ mv --backup=numbered --verbose c o
`c' -> `o' (backup: `o.~3~')

In this example, the original file o has been renamed to o.~1~, a to o.~2~, b to o.~3~, and c to o. So this doesn't rename in the same way that the code you posted does, but this may be acceptable, depending on your exact needs.

  • Far more elegant than my solution (assuming GNU mv, of course) – roaima Dec 17 '15 at 0:05

Consider two instances of your proposal running simultaneously, having started simultaneously. They will both find that (say) num=4 is available and both will attempt to use the same target name.

I can't test this alternative right now, but something like this might be sufficient

num=1 attempt=10
while ! mv --no-clobber "$1" "${PWD}/DUPES/${num}-$1" && [[ 0 -lt $attempt ]]
[[ 0 -eq $attempt ]] && echo "ERROR: Too many attempts to handle $1" >&2

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.