What's the easiest way to make a list of most common words in a list?

Question

Say I have a bunch of textfiles containing fiction, non-fiction, newspaper articles, &c (random examples of text in a given language.)

I want a frequency list of the given words, most common word first.

I could write some C code to do this, but if there's a faster way to do this, I'd like to know it. (When I say faster, I mean coding time, not run time.)

Answer 1

For faster coding time, This is what I try successfully right now :

printf '%s\n' $(cat *.txt) | sort | uniq -c | sort -gr | less 

Answer 2

Writing it in C will most likely have a fast runtime, but it takes longer to write. A good compromise might be to use awk:

tally.awk

FNR == NR { pat[$1] = 1; next }
{
  for(p in pat) 
    if(index($0, p)) 
      pat[p]++
} 
END { 
  for(p in pat) 
    print pat[p]-1, p
}

This will first read in a word list (one word per line), then count matches found in the rest of the input. Run it like this:

cat *.txt | awk -f tally.awk wordlist - | sort -nr

The above will match substrings as well, if you only want to match whole words, replace the index line with:

if(match($0, "\\<" p "\\>"))

Answer 3

You could do (with GNU grep):

grep -hEo "[[:alnum:]_'-]+" ./*.txt | sort | uniq -c | sort -rn

Change [:alnum:]_'- above to what should be word constituents.