4

I am merging two movie libraries and am looking to "de-duplicate" manually via bash scripting.

Here is my thought process so far:

  • Find all files with same name regardless of extension
  • Delete smaller file (I have storage for days! and prefer quality!)

I could build on this, so if I can somehow make the delete part separate, I can build on it. My though being I could use ffmpeg to inspect the video and pick the better one, but I'm guessing bigger size = best option and simpler to code.

I posted of Software Rec but didn't get what I wanted so I realized bash is my best bet, but my "find" knowledge is limited and most of the answers I am finding are way to complicated, I figure this should be a simple thing.

Eg: Find files with same name but different content?

1
  • I think I am making it over complicated, I'd prefer "Regardless of Extension" and I bet somebody will answer that way, but it would likely be much easier to use a list of possible extensions which I do know, eg (MKV/MP4/M4V/AVI/MPEG) Commented Apr 13, 2018 at 3:52

2 Answers 2

8

This is a nice way I wrote to just find the repeating files ignoring extension:

find . -exec bash -c 'basename "$0" ".${0##*.}"' {} \; | sort | uniq --repeated

Then I wrapped it in this loop to find the smaller of the two files for each:

for i in $(find . -exec bash -c 'basename "$0" ".${0##*.}"' {} \; | sort | uniq --repeated); do find . -name "$i*" -printf '%s %p\n' | sort -n | head -1 | cut -d ' ' -f 2-; done

Finally one more loop to (interactively, with rm -i so there's a prompt before every one) delete all those files:

for j in $(for i in $(find . -exec bash -c 'basename "$0" ".${0##*.}"' {} \; | sort | uniq --repeated); do find . -name "$i*" -printf '%s %p\n' | sort -n | head -1 | cut -d ' ' -f 2-; done); do rm -i "$j"; done

As this involves doing two finds on your directory, surely there is a better way. But this should work for simple cases. It also assumes you're working from the current directory, if you want to perform the command on a different one just change the . argument to both find commands.

1
  • 1
    The find works great, started tackeling how to compare and delete smaller file, perhaps that should have been a separate thread tho Commented Apr 13, 2018 at 4:52
1

How I ended up doing it because of my issue with needing to exclude .srt files.

Find Files with same name but different extension:

ls * | sed 's/.\{4\}$//' | sort | uniq -d

Note: This is dependant on the extension being 4 characters eg .XYZ, it wouldn't work for mpeg, but all my movies are mp4/mkv/m4v.

Ignore .srt Files:

ls * |  awk '!/.srt/'  | sed 's/.\{4\}$//' | sort | uniq -d

I posted a separate thread on comparing file sizes, but Harrys Answer does a great job except I realized I had .srt issues to mitigate.

Compare two file sizes and delete smaller file

3

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .