Preserving file extensions when renaming duplicate files with rsync

Question

I have multiple directories whose contents overlap. Sometimes the files are exactly the same, sometimes the names are the same and the contents differ and sometimes files with the same name have different contents. I merge the directories as follows:

$ rsync -aicPb --remove-source-files SOURCE-DIR/ TARGET-DIR/

This command does almost exactly what I want. The problem is that file extensions are not preserved. I.e., the -b argument causes TARGET-DIR/pictureA.jpg to be renamed to TARGET-DIR/pictureA.jpg~, when SOURCE-DIR/pictureA.jpg is to be copied to TARGET-DIR/.

Is there any way to preserve file extensions with rsync when renaming duplicate files? E.g., TARGET-DIR/pictureA.jpg should be renamed to TARGET-DIR/pictureA~.jpg.

Answer 1

You will probably have to handle this with a separate program to "fix" the filenames. Here's an example in bash to start from:

#!/bin/bash
find TARGET-DIR -name '*~' -print0 |
while IFS= read -r -d '' file
do      if [[ $file =~ \.(...)~$ ]]
        then    newname="${file%.???~}~.${BASH_REMATCH[1]}"
                if [ -e "$newname" ]
                then    echo "file exists! $newname" >&2
                else    mv -n "$file" "$newname"
                fi
        fi
done

It assumes you want to rename all files ending ~. If not, use --suffix in rsync to set a unique suffix. The filename is matched against regex \.(...)~$, namely a dot, 3 characters, and a tilde at the end. The 3 characters are captured due to the () into the bash built-in array BASH_REMATCH. The new filename is built from the old name with the .???~ glob pattern removed from the end, and the new tilde, dot, and captured extension added. -e tests if the new file already exists, else it is renamed.

Answer 2

After some further searching, I found a tool called rclone, which does exactly what I want. For those who are interested, here is the command that I use to achieve what I want:

$ rclone move --size-only --progress --fast-list --suffix `date '+.%Y%m%dT%H%M%S'` --suffix-keep-extension SOURCE-DIR/ TARGET-DIR/