One idea would be to use a combination of
- Create staging area of all files considered for the transfer using
- Delete all duplicates except for one in the staging area (only).
- Transfer the remaining files in the staging to their destination, again using
To do this, we need three locations:
- Originals, a directory path in
- Staging area, a directory path in
- Destination, a local or remote path in
First, create the staging area (this assumes that the staging area does not already exist, or if it does, that it only contains things that should be transferred):
rsync --archive --verbose --link-dest="$origdir" \
--include="*.jpg" --include="*/" --exclude="*" \
This would copy all files whose names end in
.jpg to the staging area by means of creating hard links from their original locations. Only the space to create the directory structure would be needed and the file data would not be duplicated (unless
$origdir were located on two different partitions). To add other filename patterns, add more
--include options (before the
fdupes --delete --recurse "$stagingdir"
This will interactively ask you for confirmation before removing anything. There's also a
--noprompt option that would remove the files without confirmation. Please read the
fdupes manual carefully. The files under
$origdir would not be affected by deleting files from the staging area,
Then delete empty directories from the staging directory (this is a bonus step that just cleans things up a bit):
find "$stagingdir" -type d -empty -delete -print
This would go through the entire staging area and delete any empty directory. Any deleted directory would be printed after successful deletion.
And finally transfer the non-duplicates:
rsync --archive --verbose "$stagingdir/" "$destdir"
This process would retain the original directory structure for the files that matches the patterns used in the first
rsync and that are still left in place after
fdupes has removed duplicates.