23

I had a command which would work through a text file, count all the occurrences of the words and print it out like this:

user@box $˜ magic-command-i-forgot | with grep | and awk | sort ./textfile.txt
66: the
54: and
32: I
16: unix
12: bash
5:  internet
3:  sh
1: GNU/Linux

So it does not search line-by-line, but word by word, and it does it for all the words, not just for 1 word. I'd found it somewhere on the internets a long time ago, but I cannot find or remember it..

28

I would use tr instead of awk:

echo "Lorem ipsum dolor sit sit amet et cetera." | tr '[:space:]' '[\n*]' | grep -v "^\s*$" | sort | uniq -c | sort -bnr
  • tr just replaces spaces with newlines
  • grep -v "^\s*$" trims out empty lines
  • sort to prepare as input for uniq
  • uniq -c to count occurrences
  • sort -bnr sorts in numeric reverse order while ignoring whitespace

wow. it turned out to be a great command to count swear-per-lines

find . -name "*.py" -exec cat {} \; | tr '[:space:]' '[\n*]' | grep -v "^\s*$" | sort | uniq -c | sort -bnr | grep fuck

  • Might want to use tr -s to handle multiple spaces, especially when encountering indentation. – Arcege May 21 '12 at 0:02
  • @Arcege: Good point. Although it wont change the result it may fasten script a little bit. – seler May 21 '12 at 7:10
  • -g (--general-numeric-sort) option of sort may be preferable in some cases. E.g. sort -n will keep 10\n1 4 as is, treating 1 4 as 14, while sort -g will treat it correctly as 1 4\n10. – Skippy le Grand Gourou Sep 6 '15 at 12:04
  • nice command, really deserve a vote up :) – Noor Nov 4 '15 at 13:07
7
  1. Split the input into words, one per line.
  2. Sort the resulting list of words (lines).
  3. Squash multiple occurences.
  4. Sort by occurrence count.

To split the input into words, replace any character that you deem to be a word separator by a newline.

<input_file \
tr -sc '[:alpha:]' '[\n*]' | # Add digits, -, ', ... if you consider
                             # them word constituents
sort |
uniq -c |
sort -nr
  • This is a nice answer because it can handle the case where the words are directly next to some non-word punctuation that you want to ignore. – David Grayson Mar 27 '17 at 18:09
5

Not using grep and awk but this seems to do what you want:

for w in `cat maxwell.txt`; do echo $w; done|sort|uniq -c
  2 a
  1 A
  1 an
  1 command
  1 considered
  1 domain-specific
  1 for
  1 interpreter,
  2 is
  1 language.
  1 line
  1 of
  • 1
    This won't work if the input contains shell wildcards (you need to add set -f), and treats punctuation as part of words (which can be fixed awkwardly by adding punctuation characters to IFS — good luck trying to support non-ASCII character sets). This won't be good with very large input files, as it stores the whole file in memory (sort is smarter). – Gilles May 21 '12 at 0:21
2

I believe you're after something like this?

$ perl -n -e 'foreach ${k} (split(/\s+/)){++$h{$k}};END{foreach $l (keys(%h)){print "$h{$l}: ${l}\n"}}' /path/to/your/file | sort -n -k 1

of course you can do the same with awk as well :)

2

Using awk/sort/uniq solution:

awk '{for(w=1;w<=NF;w++) print $w}' ~/textFile.txt | sort | uniq -c | sort -nr
  • Beautiful! This worked flawlessly. – stidmatt Dec 21 '18 at 5:53
0
file=/home/stefan/ooxml1.txt
for word in $(sed 's/[^A-Za-z]/ /g' $file | tr " " "\n" | sort -u)
do
  echo -n "$word "
  grep -c $word $file
done | sort -k2 -n 

sorts ascending after splitting the file into words.

The simple grep will find fish in fisheye, so you have to enhance the grep command to prevent partial matches.

Takes about 3s for a 25k text file on a machine in the ages, classical HDD (IDE).

For bigger files or often performed operations a hash map approach would be better, but for a seldom run job or smaller files only, it might be sufficient.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.