Say I have a bunch of textfiles containing fiction, non-fiction, newspaper articles, &c (random examples of text in a given language.)

I want a frequency list of the given words, most common word first.

I could write some C code to do this, but if there's a faster way to do this, I'd like to know it. (When I say faster, I mean coding time, not run time.)

  • This question is pretty close to unix.stackexchange.com/questions/41479/… Jan 21 '13 at 20:02
  • @BruceEdiger: similar but not the same, AFAIUI Korgan wants the frequency of specific words, which doesn't require any sorting.
    – Thor
    Jan 22 '13 at 9:27
  • Throw a tr [:upper:] [:lower:] into the pipe to consider This, this, THIS and even ThIs to be the same word.
    – vonbrand
    Jan 22 '13 at 20:12

For faster coding time, This is what I try successfully right now :

printf '%s\n' $(cat *.txt) | sort | uniq -c | sort -gr | less 
  • This is almost perfect, although it counts "This" and "This," as two words. Any way to fix that? Jan 21 '13 at 19:13
  • @KorganRivera, remove unwanted punctuation before passing it to printf with tr: cat *.txt | tr '[:punct:]' ' '. You may also want to make the match case-insensitive: tr 'A-Z' 'a-z'.
    – Thor
    Jan 21 '13 at 19:28
  • @Thor Thanks. I want to keep it case-sensitive. I'm trying to make a list of the most common German words in given texts. All German nouns are capitalised, so. Jan 21 '13 at 19:41
  • 2
    @Korgan Rivera - what you ask for is called "stop words" or a "stop list" See my answer in unix.stackexchange.com/questions/41479/… for how to do that. Jan 21 '13 at 20:02

Writing it in C will most likely have a fast runtime, but it takes longer to write. A good compromise might be to use awk:


FNR == NR { pat[$1] = 1; next }
  for(p in pat) 
    if(index($0, p)) 
END { 
  for(p in pat) 
    print pat[p]-1, p

This will first read in a word list (one word per line), then count matches found in the rest of the input. Run it like this:

cat *.txt | awk -f tally.awk wordlist - | sort -nr

The above will match substrings as well, if you only want to match whole words, replace the index line with:

if(match($0, "\\<" p "\\>"))

You could do (with GNU grep):

grep -hEo "[[:alnum:]_'-]+" ./*.txt | sort | uniq -c | sort -rn

Change [:alnum:]_'- above to what should be word constituents.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.