I have a text file like this:


I want to get the frequency of each word.

When I tried the following command

cat test.txt |sort|uniq -c

I got the following output

   1 america
   2 and
   1 england
   1 jerry
   1 to
   1 tom
   1 went

But I need partial matches too. ie, the word to present in the word tom. So my expected word count of to is 2. Is it possible using unix commands?


Here's one way, but it isn't very elegant:

$ sort -u file | while IFS= read -r word; do 
        printf '%s\t%s\n' "$word" "$(grep -cFe "$word" file)"; 
america 1
and 3
england 1
jerry   1
to  2
tom 1
went    1

An awk approach:

awk '
  !x {c[$0]; next}
  {for (i in c) if (index($0, i)) c[i]++}
  END{for (i in c) print c[i]"\t"i}' file x=1 file | sort -k1rn

Which on your input give

3       and
2       to
1       america
1       england
1       jerry
1       tom
1       went

We process the input in two passes. In the first pass, we record the list of distinct words as the keys of the c hash table.

In the second pass, for each line in the file, we loop over all the keys in c and increment the corresponding value if the key is found in the line.

The list of distinct words in the file ends up being stored in memory. If those are English words, it shouldn't be a problem as there are fewer and 200,000 distinct words in the English language.

  • thank you. this command works. if i run this command against a large file around 30gb, will a machine of 8gb ram handle that? – jasonroy Sep 20 at 21:11
  • @TweetMan depends how many unique words there are. It stores all unique words in memory. – Stéphane Chazelas Sep 20 at 21:14
  • Hmm. then that would be a problem. it may crash the system. – jasonroy Sep 20 at 21:20
  • Awk isn't safe with large files and it bogs down. You may want to look into loading the data into a SQL database and querying it that way. – A.Danischewski Sep 21 at 0:43
  • @A.Danischewski, awk as a language has no issue with large files. There are many implementations of an interpreter for that language. Could you please be more specific as to which implementation can't handle large files and how? How would using a SQL database help here? – Stéphane Chazelas Sep 21 at 7:41

This won't crash the system but it may take a long time to run, since it parses the input multiple times. Assuming the input file is called "in":

sort -u < in | while read w
    printf "%d\t%s\n" `grep -c "$w" in` "$w"

which on your input got me:

1       america
3       and
1       england
1       jerry
2       to
1       tom
1       went

It's not clear to me if the partial matches are to be anchored to the beginning of the line. Assuming the answer is yes, what might speed things up here is the use of binary search via the venerable look command. Of course look needs that its input file be sorted. Therefore, first create a sorted version of the original file

 sort file > file.sorted

Then loop through the original file, looking up one word at a time against the sorted file.

while read -r word; do 
    printf "%s %d\n" "$word" "$(look -b "$word" file.sorted | wc -l)"; 
done <file

Some systems don't need the -b flag to be passed to look to force a binary search. Disk caching of the sorted file could help speed things up even further

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.