3

Given a nested directory structure containing various files, I would like to find all the files within it, but where there are multiple files with the same name, I'd like to return just the largest file.

So, for example, given a directory structure like:

|--- foo.jpg (110 KB)
|--- bar.jpg (210 KB)
|--- dir
      |----- foo.jpg (860 KB)
      |----- baz.jpg (200 KB)

I'd like to produce the output lines (order unimportant):

bar.jpg
dir/foo.jpg
dir/baz.jpg

How can I do this, preferably from bash?

  • What do you need this for? Whatever you want to solve with this, I'm pretty sure there is a much cleaner solution. – michas Dec 7 '15 at 8:40
  • @michas -- my use case was that I had previously used wget to download images from a blog, but sometimes the same image had been downloaded in different resolutions, but the file name ended up the same. I wanted to deduplicate, favouring the largest copy of each. But I'm also interested in how you would solve this type of problem at the command-line. – Matt R Dec 7 '15 at 8:47
  • 1
    Vague handwave: find -printf to list all files, also printing their sizes, possibly splitting basename and dir, then sort on the size, then sort on the basename, and then see if uniq can look at single fields only or else use awk, and then reassemble dirname and basename. – Ulrich Schwarz Dec 7 '15 at 8:52
  • @UlrichSchwarz Thanks for the hint, I've fleshed it out as an answer. – Matt R Dec 7 '15 at 9:31
1

With zsh:

typeset -A files
for f (**/*(D.oL)) files[$f:t]=$f
printf '%s\n' $files

Would work whatever bytes or characters (like space, newline...) the file names may contain.

With GNU tools:

find . -type f -printf '%s/%f/%P\0' |
  sort -zrn |
  LC_ALL=C sort -zt/ -uk2,2 |
  tr '\0\n' '\n\0' |
  cut -d/ -f3- |
  tr '\0' '\n'

If you want to remove the duplicates, with zsh:

allfiles=(**/*(D.oL))
typeset -A best
for f ($allfiles) best[$f:t]=$f
bestfiles=($best)
dups=(${allfiles:|bestfiles})
rm -rf -- $dups

Description of some zsh features:

  • typeset -A best: declares an associative array variable like in ksh93. Recent versions of bash support it as well.
  • **/*: recursive globbing. Introduced by zsh in the early nineties, now found in in a few other shells with variations.
  • (D.oL): globbing qualifiers. Another zsh invention, not copied yet by other shells though it's an essential companion to recursive globbing. Used to further qualify the glob. D to include dot files, . to only include regular files, oL to order by length (size in bytes).
  • ${file:t}: like in (t)csh, expands to the tail part of a filename (the basename).
  • ${a:|b} expands to the elements of a that are not in b. (a - b).
2

Fleshing out @UlrichSchwarz's comment, I ended up with:

find . -type f -printf "%s %P %f\n" | sort -k3,3 -k1,1rn | uniq -f 2 | cut -f 2 -d ' '

Edit this won't handle filenames with (for example) spaces in. See @StéphaneChazelas' solution for something more robust.

  • I was thinking of a awk program, but your pipes make senses. – Archemar Dec 7 '15 at 9:42
  • I tried it but it does not show the expected output – user102891 Dec 7 '15 at 10:08
  • @Begueradj How so? – Matt R Dec 7 '15 at 10:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.