6

Say I have a bunch of textfiles containing fiction, non-fiction, newspaper articles, &c (random examples of text in a given language.)

I want a frequency list of the given words, most common word first.

I could write some C code to do this, but if there's a faster way to do this, I'd like to know it. (When I say faster, I mean coding time, not run time.)

3
  • This question is pretty close to unix.stackexchange.com/questions/41479/…
    – user732
    Jan 21, 2013 at 20:02
  • @BruceEdiger: similar but not the same, AFAIUI Korgan wants the frequency of specific words, which doesn't require any sorting.
    – Thor
    Jan 22, 2013 at 9:27
  • Throw a tr [:upper:] [:lower:] into the pipe to consider This, this, THIS and even ThIs to be the same word.
    – vonbrand
    Jan 22, 2013 at 20:12

3 Answers 3

4

For faster coding time, This is what I try successfully right now :

printf '%s\n' $(cat *.txt) | sort | uniq -c | sort -gr | less 
4
  • This is almost perfect, although it counts "This" and "This," as two words. Any way to fix that? Jan 21, 2013 at 19:13
  • 1
    @KorganRivera, remove unwanted punctuation before passing it to printf with tr: cat *.txt | tr '[:punct:]' ' '. You may also want to make the match case-insensitive: tr 'A-Z' 'a-z'.
    – Thor
    Jan 21, 2013 at 19:28
  • @Thor Thanks. I want to keep it case-sensitive. I'm trying to make a list of the most common German words in given texts. All German nouns are capitalised, so. Jan 21, 2013 at 19:41
  • 2
    @Korgan Rivera - what you ask for is called "stop words" or a "stop list" See my answer in unix.stackexchange.com/questions/41479/… for how to do that.
    – user732
    Jan 21, 2013 at 20:02
2

Writing it in C will most likely have a fast runtime, but it takes longer to write. A good compromise might be to use awk:

tally.awk

FNR == NR { pat[$1] = 1; next }
{
  for(p in pat) 
    if(index($0, p)) 
      pat[p]++
} 
END { 
  for(p in pat) 
    print pat[p]-1, p
}

This will first read in a word list (one word per line), then count matches found in the rest of the input. Run it like this:

cat *.txt | awk -f tally.awk wordlist - | sort -nr

The above will match substrings as well, if you only want to match whole words, replace the index line with:

if(match($0, "\\<" p "\\>"))
2

You could do (with GNU grep):

grep -hEo "[[:alnum:]_'-]+" ./*.txt | sort | uniq -c | sort -rn

Change [:alnum:]_'- above to what should be word constituents.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .