how obtain directory files info

Question

I have the need to get information about a specific directory, basically I need to know the correlation between small, medium and big files.

I came up with this:

for i in K M G; do
  printf $i
  du -h /usr/opt |
    awk '{print $1}' |
    grep ${i}$ |
    wc -l
done | tee /stat.out

from the result then I add all numbers and subtract the total to obtain the number of files under 1k. ( I presume we have a lot of them since it's source files)

Anyway, this way is good for small directories, I actually have a very big one (expecting over 1Tera) and no idea of the files distribution. I need to copy all these files to a private storage and need to give a estimate time on the copy.

I was thinking on the line of doing something as this:

find pwd |xargs ls -lph |awk '{print $5}' 

But I miss what I should put after or if I should take another approach. Any help would be most welcome.

Answer 1

If you can use GNU find (non-embedded Linux or Cygwin), make find print the file sizes and post-process the output with awk to sort each size into a category, sort and uniq to group by categories, and awk or sed to pretty-print the result. Something like:

find /usr/opt -type f -printf '%s\n' |
awk '{
    if ($1 ~ /^[2-9]......../) { print "3 G" }
    else if ($1 >= 1073741824) {  print "3 G" }
    else if ($1 >= 1048576) { print "2 M" }
    else if ($1 >= 1024) { print "1 k" }
    else if ($1 >= 1) { print "0" }
}' |
sort | uniq -c |
awk '{print $1 " files are in the " $3 "B range"}'

Answer 2

The best I came up with was to resort to a awk script.

{
if ( substr( $5, length($5), length($5) ) == "K" ) {
        totK++;
        totKsize = totKsize + substr($5, 0, length($5) - 1 );}
else if ( substr( $5, length($5), length($5) ) == "M" ) {
        totM++;
        totMsize = totMsize + substr($5, 0, length($5) - 1 );}
else if ( substr( $5, length($5), length($5) ) == "G" ) {
        totG++;
        totGsize = totGsize + substr($5, 0, length($5) - 1 );}
else  {
        totB++;
        totBsize=totBsize + $5; }
}
END{
print "NR of files less than 1k => " totB " total " totBsize;
print "NR of files less than 1M => " totK " total " totKsize;
print "NR of files less than 1G => " totM " total " totMsize;
print "NR of files bigger than 1G => " totG " total " totGsize;
}

And execute the pass like this:

find . -type f |xargs ls -lh |/usr/xpg4/bin/awk -f count_files.awk