4

I am currently trying to eliminate duplicated videos with minimal changes. Those might be a slightly different encoding, a lower resolution or just changed meta data. These videos are in a complex folder structure and therefore only the duplicate in certain folders can be deleted.

Therefore I am looking for a tool, for Linux, that just returns a list of the found duplicates, that I would then process with a simple script myself. This tool would have to be usable only with CLI (or as a library), since I would like to run it on the server directly and only let it compare certain subfolders automatically to save time (the whole server would take far too long).

The only things I found so far were dupeguru and videoduplicatefinder. Both of which are sadly GUI programs.

Currently I am using jdupes to delete all byte perfect copies. But that is too easily fooled by changed metadata or a different format like webm vs m4v vs mp4.

Does anyone know such a tool? Or is this something, I would have to write myself, if I need it?

4
  • Welcome, ffmpeg is the standard tool to use to handle videos, now I don't know if it can be used to compare videos, but maybe there's a way using some scripting. Sep 27, 2021 at 11:58
  • videoduplicatefinder requires ffmpeg which suggests it may indeed be possible to do it manually with ffmpeg.
    – terdon
    Sep 27, 2021 at 12:00
  • md5 may be a tool to compare two identical files. Sep 27, 2021 at 12:00
  • Related. Sep 27, 2021 at 12:01

1 Answer 1

3

For simple exact duplicate finder I would prefer to use fdupes tool.

You might also use your own scripts, but your description of duplicate is not strict enough to provide bullet-proof answer. For now on I would stick with ffprobe from ffmpeg package. Take a look on this command:

 ffprobe  -hide_banner -loglevel quiet -print_format json -sexagesimal -show_format -show_streams YOUR_FILE.mp4

Explanation of options:

  • -hide_banner - hides banner, we do not need it to process file
  • -log_level quiet - other messages are also not important
  • -print_format json - JSON output format allows us to use JSON processor to easily extract data we are interested in
  • -sexagesimal - time output format HH:MM:SS.MICROSECONDS
  • -show_format - gives additional info about file format
  • -show_streams - gives info about audio and video streams

This gives some interesting info. You can start with extracting some keys like format.duration using jq:

ffprobe  -hide_banner -loglevel quiet -print_format json -sexagesimal -show_format -show_streams YOUR_FILE.mp4 | jq '$.format.duration'

Identical movies should have same duration time.

After grouping them onto "possible duplicate decided by duration" directory you can take screenshot of some frame in exact moment:

ffmpeg -i YOUR_FILE.mp4 -ss 00:00:01.000 -vframes 1 output.png

This will save screenshot of whatever image there is on the beginning 1st second of movie (probably I would prefer more than one timestamps).

Then there is a matter of eventual scaling images and comparing their similiarity with ImageMagick compare (https://imagemagick.org/script/compare.php) or other similar tool

Good Luck!

1
  • For exact duplicates I am already using jdupes (very similar to fdupes, but I can tell it, to sort the results by the order of folders, I give it). So there shouldn't be any byte perfect copies left, but there are many obvious duplicates left. The features you mention seem to be good candidates to build my own fingerprints from. Similar to what findimagedupes does (which I use to maintain the image files on the same server) Sep 27, 2021 at 12:46

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .