0

How can I find count of every word in a file?

I want a histogram of each word in text pipe or document. New line and empty lines will exist in document. I stripped everything except for [a-zA-Z].

> cat doc.txt 
word second third 

word really
> cat doc.txt | ... # then count occurrences of each word \
                    # and print in descending order separated by delimiter
word 2
really 1
second 1
third 1

It needs to be somewhat efficient as file is 1GB text and cannot work with exponential time load.

6
  • Does unix.stackexchange.com/q/398413/117549 help?
    – Jeff Schaller
    Aug 12, 2020 at 14:55
  • Do you really want a fish-specific answer? I'd use perl Aug 12, 2020 at 14:58
  • @glennjackman perl is perfectly fine. fish is there to make sure it work on fish and isn't just bash/zsh specific answer as those aren't that useful for me
    – user14492
    Aug 12, 2020 at 15:26
  • Is Unix-system one word or two? What about GNU/Linux?
    – Kusalananda
    Aug 12, 2020 at 16:42
  • @Kusalananda like I said only [a-zA-Z] so hypen cannot exist; only letters small and capital case :) What about GNU/Linux. I chose to tag macOS because I want to make sure that people assume macOS flavor of tools and only expect macOS default tools to exist (installing a separate tool is overkill imo).
    – user14492
    Aug 12, 2020 at 20:22

2 Answers 2

3

Try this:

grep -o '\w*' doc.txt | sort | uniq -c | sort -nr
  • -o Print each match instead of matching lines
  • \w* Match word characters
  • sort sort the matches before piping to uniq.
  • uniq -c print the uniqe lines and the number of occurences -c
  • sort -nr Reverse sort by number of occurences.

Output:

  2 word
  1 third
  1 second
  1 really

Alternative:

Use awk for the exact output:

$ grep -o '\w*' doc.txt \
| awk '{seen[$0]++} END{for(s in seen){print s,seen[s]}}' \
| sort -k2r

word 2
really 1
second 1
third 1
1
  • With GNU awk, don't need external sort: END {PROCINFO["sorted_in"] = "@val_num_desc"; for (s in seen) print s, seen[s]} Aug 12, 2020 at 16:35
0
perl -lnE '
  $count{$_}++ for /[[:alpha:]]+/g;
  END {
    say "@$_" for
      sort {$b->[1] <=> $a->[1] || $a->[0] cmp $b->[0]}
      map {[$_, $count{$_}]}
      keys %count
  }
' doc.txt

This will consume lots more memory than pLumo's initial solution.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.