How to count occurrences of all the words in all the files of a directory using grep? But with count incremented only once per word per file

Question

I have already asked a similar question but people misunderstood what i was asking. I was asking how to generate a list of every word with a word count incremented only once per word per file.

For example, I have a directory with 10 files , I want to generate a list of words using bash commands which says a value of 1-10 depending on how many files they appear in:

10 The
10 and
8 bash
7 command
6 help....

etc.

I already know that grep -l word *| wc -l will search for a single word but I want to create a list of all words.

Is there a way to combine that with tr '[A-Z]' '[a-z]' | tr -d '[:punct:]' so that words with capital letters aren't duplicated and punctuation is removed?

Answer 1

I'd use perl here:

perl -T -lne '
  for (/\w+/g) {$count{lc $_}->{$ARGV}=undef}
  END {print "$_: " . keys %{$count{$_}} for keys %count}' ./*

That constructs a hash of hash $count{word} is a reference to a hash whose keys are the names of the files that word is found in (and values we don't care, here set to undef).

In the end, we just count the number of elements (so number of files) for each of those hashes (so for each of the found words).

Answer 2

I just came across the original answer here by @Mehmet while searching for something unrelated and I see that although it works, it is horribly inefficient, requiring each file to be read again for each unique word in all of the files! The second answer by @Jeff is rather convoluted despite the explanation and worst of all it suffers from the cat file | sin!

A single pass on all the data is all that is required and it can be formulated by effectively combining the earlier answers:

find . -maxdepth 1 -type f -print |
while read file; do
    egrep -h -o "[[:alnum:]][[:alnum:]_-]*" "$file" |
    tr '[A-Z]' '[a-z]' |
    sed "s|^|$file\||"
done |
sort -t '|' -k 2 |
uniq |
awk -F '|' '{
    if (lw != $2) {
        print fc " " lw;
        fc = 0;
    }
    lw = $2;
    fc++;
}'

Note the choice of field separator is important if your filenames include paths, and/or if they include spaces. I chose the | character since it should never be part of a word printed by egrep and it is unlikely to ever appear in a file or directory name.

Answer 3

This should get all the words from all the files, sort them and get unique words, than iterate through those words and count how many files it occurs in.

# find all words from all files within the directory
grep -o -h -E '\w+' directory/*|sort -u | \
while read word;
do
        # iterate through each word and find how many files it occurs
        c=`grep -l "$word" directory/*|wc -l`
        echo "$c $word";
done

Answer 4

This is how to process each file in a directory individually:

for f in yourdirectory/*; do cat "$f" |

This is how I filter out everything but words from text data:

sed 's/\.$//;s/\.\([^0-9]\)/\1/g;s/[][(),;:?!]//g' | tr [A-Z] [a-z] |

But your method may work just as well. (I wanted to make sure not to remove hyphens from hyphenated words, nor apostrophes from contractions.)

Either way, continue as follows:

tr -s ' ' '\012' | sort -u ; done |

This makes a one-per-file list of words, so now just:

sort | uniq -c

If you want the list from most- to least-frequent, just tack on |sort -nr.

You may also need to add some additional punctuation, such as {} to the list at the end of the sed above, depending on your input data.