Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Say I have a bunch of textfiles containing fiction, non-fiction, newspaper articles, &c (random examples of text in a given language.)

I want a frequency list of the given words, most common word first.

I could write some C code to do this, but if there's a faster way to do this, I'd like to know it. (When I say faster, I mean coding time, not run time.)

share|improve this question
    
This question is pretty close to unix.stackexchange.com/questions/41479/… –  Bruce Ediger Jan 21 '13 at 20:02
    
@BruceEdiger: similar but not the same, AFAIUI Korgan wants the frequency of specific words, which doesn't require any sorting. –  Thor Jan 22 '13 at 9:27
    
Throw a tr [:upper:] [:lower:] into the pipe to consider This, this, THIS and even ThIs to be the same word. –  vonbrand Jan 22 '13 at 20:12
add comment

3 Answers 3

For faster coding time, This is what I try successfully right now :

printf '%s\n' $(cat *.txt) | sort | uniq -c | sort -gr | less 
share|improve this answer
    
This is almost perfect, although it counts "This" and "This," as two words. Any way to fix that? –  Korgan Rivera Jan 21 '13 at 19:13
    
@KorganRivera, remove unwanted punctuation before passing it to printf with tr: cat *.txt | tr '[:punct:]' ' '. You may also want to make the match case-insensitive: tr 'A-Z' 'a-z'. –  Thor Jan 21 '13 at 19:28
    
@Thor Thanks. I want to keep it case-sensitive. I'm trying to make a list of the most common German words in given texts. All German nouns are capitalised, so. –  Korgan Rivera Jan 21 '13 at 19:41
2  
@Korgan Rivera - what you ask for is called "stop words" or a "stop list" See my answer in unix.stackexchange.com/questions/41479/… for how to do that. –  Bruce Ediger Jan 21 '13 at 20:02
add comment

Writing it in C will most likely have a fast runtime, but it takes longer to write. A good compromise might be to use awk:

tally.awk

FNR == NR { pat[$1] = 1; next }
{
  for(p in pat) 
    if(index($0, p)) 
      pat[p]++
} 
END { 
  for(p in pat) 
    print pat[p]-1, p
}

This will first read in a word list (one word per line), then count matches found in the rest of the input. Run it like this:

cat *.txt | awk -f tally.awk wordlist - | sort -nr

The above will match substrings as well, if you only want to match whole words, replace the index line with:

if(match($0, "\\<" p "\\>"))
share|improve this answer
add comment

You could do (with GNU grep):

grep -hEo "[[:alnum:]_'-]+" ./*.txt | sort | uniq -c | sort -rn

Change [:alnum:]_'- above to what should be word constituents.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.