Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I did a website scrape for a conversion project. I'd like to do some statistics on the types of files in there -- for instance, 400 .html files, 100 .gif, etc. What's an easy way to do this? It has to be recursive.

Edit: With the script that maxschelpzig posted, I'm having some problems due to the architecture of the site I've scraped. Some of the files are of the name *.php?blah=blah&foo=bar with various arguments, so it counts them all as unique. So the solution needs to consider *.php* to be all of the same type, so to speak.

share|improve this question

2 Answers 2

up vote 15 down vote accepted

You could use find and uniq for this, e.g.:

$ find . -type f | sed 's/.*\.//' | sort | uniq -c
   16 avi
   29 jpg
  136 mp3
    3 mp4

Command explanation

  • find recursively prints all filenames
  • sed deletes from every filename the prefix until the file extension
  • uniq assumes sorted input
    • -c does the counting (like a histogram).
share|improve this answer
    
I have a similar script. Simple and fast. –  J.F. Aug 10 '11 at 19:10
    
Some of the files are of the name *.php?blah=blah&foo=bar with various arguments, so it counts them all as unique. How can I modify it to look for *.php*? –  user394 Aug 11 '11 at 13:35
2  
You can try to use a different sed expression, e.g. sed 's/^.*\(\.[a-zA-Z0-9][a-zA-Z0-9][a-zA-Z0-9]\).*$/\1/' –  maxschlepzig Aug 11 '11 at 14:35
    
@maxschlepzig awesome, thanks so much! :) –  user394 Aug 11 '11 at 16:04
    
Most useful is this - thanks! –  danherd Feb 17 '12 at 9:26

With zsh:

print -rl **/?*.*(.:e) | uniq -c |sort -k1n

The pattern **/?*.* matches all files that have an extension, in the current directory and its subdirectories recursively. The glob qualifier . selects only regular files. The history modifier retains only the file extension. print -rl prints one match per line. uniq -c counts consecutive identical items (the glob result is already sorted). The final call to sort sorts the extensions by use count.

share|improve this answer
    
This works awesome –  JackLeo Sep 6 '11 at 16:59

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.