Is there a fuzzy duplicate finder for videos, that does not require a GUI?

Question

I am currently trying to eliminate duplicated videos with minimal changes. Those might be a slightly different encoding, a lower resolution or just changed meta data. These videos are in a complex folder structure and therefore only the duplicate in certain folders can be deleted.

Therefore I am looking for a tool, for Linux, that just returns a list of the found duplicates, that I would then process with a simple script myself. This tool would have to be usable only with CLI (or as a library), since I would like to run it on the server directly and only let it compare certain subfolders automatically to save time (the whole server would take far too long).

The only things I found so far were dupeguru and videoduplicatefinder. Both of which are sadly GUI programs.

Currently I am using jdupes to delete all byte perfect copies. But that is too easily fooled by changed metadata or a different format like webm vs m4v vs mp4.

Does anyone know such a tool? Or is this something, I would have to write myself, if I need it?

Answer 1

For simple exact duplicate finder I would prefer to use fdupes tool.

You might also use your own scripts, but your description of duplicate is not strict enough to provide bullet-proof answer. For now on I would stick with ffprobe from ffmpeg package. Take a look on this command:

 ffprobe  -hide_banner -loglevel quiet -print_format json -sexagesimal -show_format -show_streams YOUR_FILE.mp4

Explanation of options:

• -hide_banner - hides banner, we do not need it to process file
• -log_level quiet - other messages are also not important
• -print_format json - JSON output format allows us to use JSON processor to easily extract data we are interested in
• -sexagesimal - time output format HH:MM:SS.MICROSECONDS
• -show_format - gives additional info about file format
• -show_streams - gives info about audio and video streams

This gives some interesting info. You can start with extracting some keys like format.duration using jq:

ffprobe  -hide_banner -loglevel quiet -print_format json -sexagesimal -show_format -show_streams YOUR_FILE.mp4 | jq '$.format.duration'

Identical movies should have same duration time.

After grouping them onto "possible duplicate decided by duration" directory you can take screenshot of some frame in exact moment:

ffmpeg -i YOUR_FILE.mp4 -ss 00:00:01.000 -vframes 1 output.png

This will save screenshot of whatever image there is on the beginning 1st second of movie (probably I would prefer more than one timestamps).

Then there is a matter of eventual scaling images and comparing their similiarity with ImageMagick compare (https://imagemagick.org/script/compare.php) or other similar tool

Good Luck!