3

Given a nested directory structure containing various files, I would like to find all the files within it, but where there are multiple files with the same name, I'd like to return just the largest file.

So, for example, given a directory structure like:

|--- foo.jpg (110 KB)
|--- bar.jpg (210 KB)
|--- dir
      |----- foo.jpg (860 KB)
      |----- baz.jpg (200 KB)

I'd like to produce the output lines (order unimportant):

bar.jpg
dir/foo.jpg
dir/baz.jpg

How can I do this, preferably from bash?

4
  • What do you need this for? Whatever you want to solve with this, I'm pretty sure there is a much cleaner solution.
    – michas
    Dec 7, 2015 at 8:40
  • @michas -- my use case was that I had previously used wget to download images from a blog, but sometimes the same image had been downloaded in different resolutions, but the file name ended up the same. I wanted to deduplicate, favouring the largest copy of each. But I'm also interested in how you would solve this type of problem at the command-line.
    – Matt R
    Dec 7, 2015 at 8:47
  • 1
    Vague handwave: find -printf to list all files, also printing their sizes, possibly splitting basename and dir, then sort on the size, then sort on the basename, and then see if uniq can look at single fields only or else use awk, and then reassemble dirname and basename. Dec 7, 2015 at 8:52
  • @UlrichSchwarz Thanks for the hint, I've fleshed it out as an answer.
    – Matt R
    Dec 7, 2015 at 9:31

2 Answers 2

2

Fleshing out @UlrichSchwarz's comment, I ended up with:

find . -type f -printf "%s %P %f\n" | sort -k3,3 -k1,1rn | uniq -f 2 | cut -f 2 -d ' '

Edit this won't handle filenames with (for example) spaces in. See @StéphaneChazelas' solution for something more robust.

3
  • I was thinking of a awk program, but your pipes make senses.
    – Archemar
    Dec 7, 2015 at 9:42
  • I tried it but it does not show the expected output
    – user102891
    Dec 7, 2015 at 10:08
  • @Begueradj How so?
    – Matt R
    Dec 7, 2015 at 10:31
1

With zsh:

typeset -A files
for f (**/*(D.oL)) files[$f:t]=$f
printf '%s\n' $files

Would work whatever bytes or characters (like space, newline...) the file names may contain.

With GNU tools:

find . -type f -printf '%s/%f/%P\0' |
  sort -zrn |
  LC_ALL=C sort -zt/ -uk2,2 |
  tr '\0\n' '\n\0' |
  cut -d/ -f3- |
  tr '\0' '\n'

If you want to remove the duplicates, with zsh:

allfiles=(**/*(D.oL))
typeset -A best
for f ($allfiles) best[$f:t]=$f
bestfiles=($best)
dups=(${allfiles:|bestfiles})
rm -rf -- $dups

Description of some zsh features:

  • typeset -A best: declares an associative array variable like in ksh93. Recent versions of bash support it as well.
  • **/*: recursive globbing. Introduced by zsh in the early nineties, now found in in a few other shells with variations.
  • (D.oL): globbing qualifiers. Another zsh invention, not copied yet by other shells though it's an essential companion to recursive globbing. Used to further qualify the glob. D to include dot files, . to only include regular files, oL to order by length (size in bytes).
  • ${file:t}: like in (t)csh, expands to the tail part of a filename (the basename).
  • ${a:|b} expands to the elements of a that are not in b. (a - b).

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .