Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I had a command which would work through a text file, count all the occurrences of the words and print it out like this:

remy@box $˜ magic-command-i-forgot | with grep | and awk | sort ./textfile.txt
66: the
54: and
32: I
16: unix
12: bash
5:  internet
3:  sh
1: GNU/Linux

So it does not search line-by-line, but word by word, and it does it for all the words, not just for 1 word. I'd found it somewhere on the internets a long time ago, but I cannot find or remember it..

share|improve this question
add comment

6 Answers

up vote 4 down vote accepted

I would use tr instead of awk:

echo "Lorem ipsum dolor sit sit amet." | tr [:space:] '\n' | grep -v "^\s*$" | sort | uniq -c | sort -bnr
  • tr just replaces spaces with newlines
  • grep -v "^\s*$" trims out empty lines
  • sort to prepare as input for uniq
  • uniq -c to count occurrences
  • sort -bnr sorts in numeric reverse order while ignoring whitespace

wow. it turned out to be a great command to count swear-per-lines

find . -name "*.py" -exec cat {} \; | tr [:space:] '\n' | grep -v "^\s*$" | sort | uniq -c | sort -bnr | grep fuck

share|improve this answer
    
Might want to use tr -s to handle multiple spaces, especially when encountering indentation. –  Arcege May 21 '12 at 0:02
    
@Arcege: Good point. Although it wont change the result it may fasten script a little bit. –  seler May 21 '12 at 7:10
    
Thanks! Liked the kernel "word" counter. –  Raymii May 22 '12 at 4:07
add comment
  1. Split the input into words, one per line.
  2. Sort the resulting list of words (lines).
  3. Squash multiple occurences.
  4. Sort by occurrence count.

To split the input into words, replace any character that you deem to be a word separator by a newline.

<input_file \
tr -sc '[:alpha:]' '\n' |   # Add digits, -, \', ... if you consider them word constituents
sort |
uniq -c |
sort -k 1nr
share|improve this answer
add comment

I believe you're after something like this?

$ perl -n -e 'foreach ${k} (split(/\s+/)){++$h{$k}};END{foreach $l (keys(%h)){print "$h{$l}: ${l}\n"}}' /path/to/your/file | sort -n -k 1

of course you can do the same with awk as well :)

share|improve this answer
add comment

Not using grep and awk but this seems to do what you want:

for w in `cat maxwell.txt`; do echo $w; done|sort|uniq -c
  2 a
  1 A
  1 an
  1 command
  1 considered
  1 domain-specific
  1 for
  1 interpreter,
  2 is
  1 language.
  1 line
  1 of
share|improve this answer
    
This won't work if the input contains shell wildcards (you need to add set -f), and treats punctuation as part of words (which can be fixed awkwardly by adding punctuation characters to IFS — good luck trying to support non-ASCII character sets). This won't be good with very large input files, as it stores the whole file in memory (sort is smarter). –  Gilles May 21 '12 at 0:21
add comment

Using awk/sort/uniq solution:

awk '{for(w=1;w<=NF;w++) print $w}' ~/textFile.txt | sort | uniq -c | sort -nr
share|improve this answer
add comment
file=/home/stefan/ooxml1.txt
for word in $(sed 's/[^A-Za-z]/ /g' $file | tr " " "\n" | sort -u)
do
  echo -n "$word "
  grep -c $word $file
done | sort -k2 -n 

sorts ascending after splitting the file into words.

The simple grep will find fish in fisheye, so you have to enhance the grep command to prevent partial matches.

Takes about 3s for a 25k text file on a machine in the ages, classical HDD (IDE).

For bigger files or often performed operations a hash map approach would be better, but for a seldom run job or smaller files only, it might be sufficient.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.