How to get the distribution of file sizes?

Question

I'd like to know the distribution of file sizes under a certain directory.

Please note: distribution of file sizes, not size of a directory. That means I want to know there are 25 files of 60 bytes, 50 files of 12587 bytes, 2 files of 57kbytes, and so on.

Bonus points if the data could be gathered via command line (eg. on a remote system) in a format easily useable to produce graphs.

Answer 1

List the files, extract the size in bytes from the list, sort it and count the occurrence of every size:

find /my/directory -type f -exec ls -l {} + | cut -d' ' -f5 | sort -n | uniq -c

• not terribly efficient
• if there are many many files it may be better to save intermediate results in a temp file, sort it to another temp file, then "uniq" it
• here I use numeric sort so the output will be ordered by ascending file size (nice), but any sort will do as long as equal lines are grouped together
• pipe the results in awk '{ print $1 "," $2 }' to get a CSV file to be used in your graphing tool of choice (even spreasheet tools will do)

Answer 2

A variant of Luke404's with GNU find:

find . -type f -printf '%s\n' | sort -n | uniq -c

Answer 3

This Perl code may help:

@files = grep {-f} glob "*"; #List files in the current directory   
%files;  
for(@files)  
{  
  chomp (my $size = \`du -h \"$_\"`);  
  $size=~ s/\s+.*//;  #Remove the file name from the output of du  
  $files{$size}++;  #  Add an entry to the hash  
}  
print "Size,Count\n"; # Print a header  
print "${\_},$files{$_}\n" for(keys %files); # Print info in CSV format  

Note the following:

• This code does not attempt to sort the files by size (this would probably need a subroutine by itself)
• I'm using du rather than the -s operator of Perl in order to have human readable output.
• If you want to list the contents of a directory other than the current one, replace glob "*" by glob "$ARGV[0]/*" and supply the name of your required directory as a command line argument.

• If you want to list the contents of several directories in a batch job, you can save this code as list_dir.pl and have a bash wrapper that does something like this:

  list_dir.pl dir1 > dir1_list.csv  
  list_dir.pl dir2 > dir2_list.csv
  

• Alternatively, the code can be further modified to accept several directories as arguments and fork a process for each directory it's searching.

I'm assuming you want files in the current directory only one level deep (no recursion).