7

I work on a cluster shared with other colleagues. The hard disk is limited (and has been full on some occasions), so I clean up my part occasionally. I want to do this quickly, so until now I do this by making a list of files larger than 100 MB older than 3 months, and I see if I still need them.

But now I am thinking that there could be a folder with >1000 smaller files that I miss, so I want to get an easy way to see if this is the case. From the way I generate data, it would help to get a list of total size per extension. In the context of this question, 'extension' as everything behind the last dot in the filename.

Suppose I have multiple folders with multiple files:

folder1/file1.bmp   40 kiB
folder1/file2.jpg   20 kiB
folder2/file3.bmp   30 kiB
folder2/file4.jpg    8 kiB

Is it possible to make a list of total filesize per file extension, so like this:

bmp 70 kiB
jpg 28 kiB

I don't care about files without extension, so they can be ignored or put in one category.

I already went through man pages of ls, du and find, but I don't know what is the right tool for this job...

9

On a GNU system:

find . -name '?*.*' -type f -printf '%b.%f\0' |
  awk -F . -v RS='\0' '
    {s[$NF] += $1; n[$NF]++}
    END {for (e in s) printf "%15d %4d %s\n", s[e]*512, n[e], e}' |
  sort -n

Or the same with perl, avoiding the -printf extension of GNU find (still using a GNU extension, -print0, but this one is more widely supported nowadays):

find . -name '?*.*' -type f -print0 |
  perl -0ne '
    if (@s = stat$_){
      ($ext = $_) =~ s/.*\.//s;
      $s{$ext} += $s[12];
      $n{$ext}++;
    }
    END {
      for (sort{$s{$a} <=> $s{$b}} keys %s) {
        printf "%15d %4d %s\n",  $s{$_}<<9, $n{$_}, $_;
      }
    }'

It gives an output like:

          12288    1 pnm
          16384    4 gif
         204800    2 ico
        1040384   17 jpg
        2752512   83 png

If you want KiB, MiB... suffixes, pipe to numfmt --to=iec-i --suffix=B.

%b*512 gives the disk usage, but note that if files are hard linked several times, they will be counted several times so you may see a discrepancy with what du reports.

  • This code does what I want, great! – Pakk Sep 9 '16 at 11:17
  • Fails on MacOS (find: -printf: unknown primary or operator) – MichaelCodes Apr 11 at 15:45
  • 1
    @MichaelCodes, yes -printf is specific to GNU find, which is why I said on a GNU system. – Stéphane Chazelas Apr 11 at 15:50
  • @MichaelCodes, see edit with a perl alternative that should work even on macOS. – Stéphane Chazelas Apr 11 at 16:12
2

Here is another solution:

find . -type f |  egrep -o "\.[a-zA-Z0-9]+$" | sort -u | xargs -I '%' find . -type f -name "*%" -exec du -ch {} + -exec echo % \; | egrep "^\.[a-zA-Z0-9]+$|total$" | uniq | paste - -

The part that gets the extensions is:

find . -type f |  egrep -o "\.[a-zA-Z0-9]+$" | sort -u

Next search for the files with an extension and print it on the screen as well:

xargs -I '%' find . -type f -name "*%" -exec du -ch {} + -exec echo % \;

Next we want to keep the extension and the total:

egrep "^\.[a-zA-Z0-9]+$|total$" | uniq

and keep it on the same line:

paste - -
1

Not as nice as Stephane's solution, but you could try

find . -type f -name "*.png" -print0 | xargs -0r du -ch | tail -n1

where you have to run this for each type of files.

  • That assumes there are sufficiently few png files that only one du invocation is run. With GNU xargs, you'd want to add the -r flag so du is not run when there's no file (otherwise, you'd end up with the disk usage of the current directory). You may want to add a -type f or ! type d to avoid counting the files that are in directories whose name ends in .png. – Stéphane Chazelas Sep 9 '16 at 11:02
  • this only looks for one specific extension. – Rahul Sep 9 '16 at 11:14
  • That's what I wrote. One had to wrap it in a script that iterates over all applicable extensions in order to get a "complete" solution. – countermode Sep 9 '16 at 11:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.