Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It only takes a minute to sign up.
Sign up to join this community
I had a command which would work through a text file, count all the occurrences of the words and print it out like this:
user@box $˜ magic-command-i-forgot | with grep | and awk | sort ./textfile.txt
66: the
54: and
32: I
16: unix
12: bash
5: internet
3: sh
1: GNU/Linux
So it does not search line-by-line, but word by word, and it does it for all the words, not just for 1 word. I'd found it somewhere on the internets a long time ago, but I cannot find or remember it..
I would use tr instead of awk:
echo "Lorem ipsum dolor sit sit amet et cetera." | tr '[:space:]' '[\n*]' | grep -v "^\s*$" | sort | uniq -c | sort -bnr
tr just replaces spaces with newlinesgrep -v "^\s*$" trims out empty linessort to prepare as input for uniquniq -c to count occurrencessort -bnr sorts in numeric reverse order while ignoring whitespace wow. it turned out to be a great command to count swear-per-lines
find . -name "*.py" -exec cat {} \; | tr '[:space:]' '[\n*]' | grep -v "^\s*$" | sort | uniq -c | sort -bnr | grep fuck
tr -s to handle multiple spaces, especially when encountering indentation.
-g (--general-numeric-sort) option of sort may be preferable in some cases. E.g. sort -n will keep 10\n1 4 as is, treating 1 4 as 14, while sort -g will treat it correctly as 1 4\n10.
Sep 6, 2015 at 12:04
echo "Lorem ipsum dolor sit sit amet et cetera." | tr ' ' '\n' | grep -v "^$" | sort | uniq -c | sort -bnr Note that I am a beginner, so I may be wrong, please feel free to advise.
To split the input into words, replace any character that you deem to be a word separator by a newline.
<input_file \
tr -sc '[:alpha:]' '[\n*]' | # Add digits, -, ', ... if you consider
# them word constituents
sort |
uniq -c |
sort -nr
Not using grep and awk but this seems to do what you want:
for w in `cat maxwell.txt`; do echo $w; done|sort|uniq -c
2 a
1 A
1 an
1 command
1 considered
1 domain-specific
1 for
1 interpreter,
2 is
1 language.
1 line
1 of
set -f), and treats punctuation as part of words (which can be fixed awkwardly by adding punctuation characters to IFS — good luck trying to support non-ASCII character sets). This won't be good with very large input files, as it stores the whole file in memory (sort is smarter).
May 21, 2012 at 0:21
Using awk/sort/uniq solution:
awk '{for(w=1;w<=NF;w++) print $w}' ~/textFile.txt | sort | uniq -c | sort -nr
I believe you're after something like this?
$ perl -n -e 'foreach ${k} (split(/\s+/)){++$h{$k}};END{foreach $l (keys(%h)){print "$h{$l}: ${l}\n"}}' /path/to/your/file | sort -n -k 1
of course you can do the same with awk as well :)
file=/home/stefan/ooxml1.txt
for word in $(sed 's/[^A-Za-z]/ /g' $file | tr " " "\n" | sort -u)
do
echo -n "$word "
grep -c $word $file
done | sort -k2 -n
sorts ascending after splitting the file into words.
The simple grep will find fish in fisheye, so you have to enhance the grep command to prevent partial matches.
Takes about 3s for a 25k text file on a machine in the ages, classical HDD (IDE).
For bigger files or often performed operations a hash map approach would be better, but for a seldom run job or smaller files only, it might be sufficient.
