Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have the need to get information about a specific directory, basically I need to know the correlation between small, medium and big files.

I came up with this:

for i in K M G; do
  printf $i
  du -h /usr/opt |
    awk '{print $1}' |
    grep ${i}$ |
    wc -l
done | tee /stat.out

from the result then I add all numbers and subtract the total to obtain the number of files under 1k. ( I presume we have a lot of them since it's source files)

Anyway, this way is good for small directories, I actually have a very big one (expecting over 1Tera) and no idea of the files distribution. I need to copy all these files to a private storage and need to give a estimate time on the copy.

I was thinking on the line of doing something as this:

find pwd |xargs ls -lph |awk '{print $5}' 

But I miss what I should put after or if I should take another approach. Any help would be most welcome.

share|improve this question
    
is there a reason you aren't using find ... -size -1k? –  h3rrmiller Feb 19 '13 at 15:14
    
that forces me to do as well several passes for each size that I need. which takes ages. :/ I would like to to it in one pass of find instead of 4. that's why I'm using find with ls, this way the bytes are shown as numbers, and the rest with numbers and size: examples 123K, 84M, etc. –  BitsOfNix Feb 19 '13 at 15:26
    
Just collect the data in one ls, and process the list of sizes? Could even do fancy histograms and other pretty images... –  vonbrand Feb 19 '13 at 15:46
add comment

2 Answers 2

up vote 0 down vote accepted

If you can use GNU find (non-embedded Linux or Cygwin), make find print the file sizes and post-process the output with awk to sort each size into a category, sort and uniq to group by categories, and awk or sed to pretty-print the result. Something like:

find /usr/opt -type f -printf '%s\n' |
awk '{
    if ($1 ~ /^[2-9]......../) { print "3 G" }
    else if ($1 >= 1073741824) {  print "3 G" }
    else if ($1 >= 1048576) { print "2 M" }
    else if ($1 >= 1024) { print "1 k" }
    else if ($1 >= 1) { print "0" }
}' |
sort | uniq -c |
awk '{print $1 " files are in the " $3 "B range"}'
share|improve this answer
    
Thanks, a lot cleaner than what I had! –  BitsOfNix Mar 27 '13 at 2:43
add comment

The best I came up with was to resort to a awk script.

{
if ( substr( $5, length($5), length($5) ) == "K" ) {
        totK++;
        totKsize = totKsize + substr($5, 0, length($5) - 1 );}
else if ( substr( $5, length($5), length($5) ) == "M" ) {
        totM++;
        totMsize = totMsize + substr($5, 0, length($5) - 1 );}
else if ( substr( $5, length($5), length($5) ) == "G" ) {
        totG++;
        totGsize = totGsize + substr($5, 0, length($5) - 1 );}
else  {
        totB++;
        totBsize=totBsize + $5; }
}
END{
print "NR of files less than 1k => " totB " total " totBsize;
print "NR of files less than 1M => " totK " total " totKsize;
print "NR of files less than 1G => " totM " total " totMsize;
print "NR of files bigger than 1G => " totG " total " totGsize;
}

And execute the pass like this:

find . -type f |xargs ls -lh |/usr/xpg4/bin/awk -f count_files.awk
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.