Finding duplicate files and moving ONE copy to another drive, deleting all other copies

Question

Actually I am trying to do two different things....

First off is to make a copy (onto my travel HDD) of all the video files, photos and documents, excluding any duplicates. So there is only one copy on my travel HDD. This would have to be able to see into many levels of folders (ntfs file system). Second thing is to transfer one copy of all those files while deleting from source any duplicates, leaving one original in source and the copy in the travel drive. This is for two different systems, that is why the two similar requests

I would like to be able to limit it to any particular file type (either by encoding comparison or by .xyz extension) as well as making sure the files are hash checked for duplication.

My needs combine the duplicate file finding function with the automated transfer/copy onto another media... preferably all in one step.

Is there such a tool out there? or how would I use the command line to do this?

Answer 1

One idea would be to use a combination of fdupes and rsync.

1. Create staging area of all files considered for the transfer using rsync.
2. Delete all duplicates except for one in the staging area (only).
3. Transfer the remaining files in the staging to their destination, again using rsync.

To do this, we need three locations:

1. Originals, a directory path in $origdir.
2. Staging area, a directory path in $stagingdir.
3. Destination, a local or remote path in $destdir.

First, create the staging area (this assumes that the staging area does not already exist, or if it does, that it only contains things that should be transferred):

rsync --archive --verbose --link-dest="$origdir" \
    --include="*.jpg" --include="*/" --exclude="*" \
    "$origdir/" "$stagingdir"

This would copy all files whose names end in .jpg to the staging area by means of creating hard links from their original locations. Only the space to create the directory structure would be needed and the file data would not be duplicated (unless $stagingdir and $origdir were located on two different partitions). To add other filename patterns, add more --include options (before the --exclude).

Then run fdupes over $stagingdir:

fdupes --delete --recurse "$stagingdir"

This will interactively ask you for confirmation before removing anything. There's also a --noprompt option that would remove the files without confirmation. Please read the fdupes manual carefully. The files under $origdir would not be affected by deleting files from the staging area,

Then delete empty directories from the staging directory (this is a bonus step that just cleans things up a bit):

find "$stagingdir" -type d -empty -delete -print

This would go through the entire staging area and delete any empty directory. Any deleted directory would be printed after successful deletion.

And finally transfer the non-duplicates:

rsync --archive --verbose "$stagingdir/" "$destdir"

This process would retain the original directory structure for the files that matches the patterns used in the first rsync and that are still left in place after fdupes has removed duplicates.