5

I have already asked a similar question but people misunderstood what i was asking. I was asking how to generate a list of every word with a word count incremented only once per word per file.

For example, I have a directory with 10 files , I want to generate a list of words using bash commands which says a value of 1-10 depending on how many files they appear in:

10 The
10 and
8 bash
7 command
6 help....

etc.

I already know that grep -l word *| wc -l will search for a single word but I want to create a list of all words.

Is there a way to combine that with tr '[A-Z]' '[a-z]' | tr -d '[:punct:]' so that words with capital letters aren't duplicated and punctuation is removed?

  • 1
    Instead of cross posting questions1 and 2 please edit your question and make it clear. – αғsнιη Dec 14 '14 at 13:34
  • Sorry but this is the only way I know how to explain it, I don't know how i can make it clearer. I'm looking for something that can count, not how many times the word occurs in a directory but how many files it occurs in. – Cnvrsn Dec 14 '14 at 13:37
  • So none of the given answers worked for you? 7 answers – αғsнιη Dec 14 '14 at 13:39
  • Nope, unfortunately not, i've tried clarifying it to the people who have tried answering it but to no avail – Cnvrsn Dec 14 '14 at 13:40
  • 1
    Why "using grep"? Have you considered, for example, awk? – steeldriver Dec 14 '14 at 20:25
2

I'd use perl here:

perl -T -lne '
  for (/\w+/g) {$count{lc $_}->{$ARGV}=undef}
  END {print "$_: " . keys %{$count{$_}} for keys %count}' ./*

That constructs a hash of hash $count{word} is a reference to a hash whose keys are the names of the files that word is found in (and values we don't care, here set to undef).

In the end, we just count the number of elements (so number of files) for each of those hashes (so for each of the found words).

1

I just came across the original answer here by @Mehmet while searching for something unrelated and I see that although it works, it is horribly inefficient, requiring each file to be read again for each unique word in all of the files! The second answer by @Jeff is rather convoluted despite the explanation and worst of all it suffers from the cat file | sin!

A single pass on all the data is all that is required and it can be formulated by effectively combining the earlier answers:

find . -maxdepth 1 -type f -print |
while read file; do
    egrep -h -o "[[:alnum:]][[:alnum:]_-]*" "$file" |
    tr '[A-Z]' '[a-z]' |
    sed "s|^|$file\||"
done |
sort -t '|' -k 2 |
uniq |
awk -F '|' '{
    if (lw != $2) {
        print fc " " lw;
        fc = 0;
    }
    lw = $2;
    fc++;
}'

Note the choice of field separator is important if your filenames include paths, and/or if they include spaces. I chose the | character since it should never be part of a word printed by egrep and it is unlikely to ever appear in a file or directory name.

-1

This should get all the words from all the files, sort them and get unique words, than iterate through those words and count how many files it occurs in.

# find all words from all files within the directory
grep -o -h -E '\w+' directory/*|sort -u | \
while read word;
do
        # iterate through each word and find how many files it occurs
        c=`grep -l "$word" directory/*|wc -l`
        echo "$c $word";
done
  • Yes that works for all word occurrences but i'm looking for occurrence PER FILE. So each word has a number next to it stating how many files it appears in, not how many times it occurs in those files – Cnvrsn Dec 14 '14 at 13:35
  • I see @Cnvrsn I updated it, take a look. – Mehmet Dec 14 '14 at 13:45
-1

This is how to process each file in a directory individually:

for f in yourdirectory/*; do cat "$f" |

This is how I filter out everything but words from text data:

sed 's/\.$//;s/\.\([^0-9]\)/\1/g;s/[][(),;:?!]//g' | tr [A-Z] [a-z] |

But your method may work just as well. (I wanted to make sure not to remove hyphens from hyphenated words, nor apostrophes from contractions.)

Either way, continue as follows:

tr -s ' ' '\012' | sort -u ; done |

This makes a one-per-file list of words, so now just:

sort | uniq -c

If you want the list from most- to least-frequent, just tack on |sort -nr.

You may also need to add some additional punctuation, such as {} to the list at the end of the sed above, depending on your input data.

  • 1
    Congratulations, you win a UUOC. – Tom Zych Dec 20 '15 at 23:14
  • Yeah, I know, but it made the possible substitution of OP code (for the second block) less fraught. – Jeff Y Dec 20 '15 at 23:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.