Find files recursively, but choose largest from among those with duplicate names

Question

Given a nested directory structure containing various files, I would like to find all the files within it, but where there are multiple files with the same name, I'd like to return just the largest file.

So, for example, given a directory structure like:

|--- foo.jpg (110 KB)
|--- bar.jpg (210 KB)
|--- dir
      |----- foo.jpg (860 KB)
      |----- baz.jpg (200 KB)

I'd like to produce the output lines (order unimportant):

bar.jpg
dir/foo.jpg
dir/baz.jpg

How can I do this, preferably from bash?

Answer 1

Fleshing out @UlrichSchwarz's comment, I ended up with:

find . -type f -printf "%s %P %f\n" | sort -k3,3 -k1,1rn | uniq -f 2 | cut -f 2 -d ' '

Edit this won't handle filenames with (for example) spaces in. See @StéphaneChazelas' solution for something more robust.

Answer 2

With zsh:

typeset -A files
for f (**/*(D.oL)) files[$f:t]=$f
printf '%s\n' $files

Would work whatever bytes or characters (like space, newline...) the file names may contain.

With GNU tools:

find . -type f -printf '%s/%f/%P\0' |
  sort -zrn |
  LC_ALL=C sort -zt/ -uk2,2 |
  tr '\0\n' '\n\0' |
  cut -d/ -f3- |
  tr '\0' '\n'

If you want to remove the duplicates, with zsh:

allfiles=(**/*(D.oL))
typeset -A best
for f ($allfiles) best[$f:t]=$f
bestfiles=($best)
dups=(${allfiles:|bestfiles})
rm -rf -- $dups

Description of some zsh features:

• typeset -A best: declares an associative array variable like in ksh93. Recent versions of bash support it as well.
• **/*: recursive globbing. Introduced by zsh in the early nineties, now found in in a few other shells with variations.
• (D.oL): globbing qualifiers. Another zsh invention, not copied yet by other shells though it's an essential companion to recursive globbing. Used to further qualify the glob. D to include dot files, . to only include regular files, oL to order by length (size in bytes).
• ${file:t}: like in (t)csh, expands to the tail part of a filename (the basename).
• ${a:|b} expands to the elements of a that are not in b. (a - b).