0

I am trying to count the number of each word in a txt file.

words.txt

the day is sunny the the
the sunny is is

expected result:

the 4
is 3
sunny 2
day 1

I want to

1) replace all spaces with new line

awk -v RS=" " '{print}' words.txt

result

the
day
is
sunny
the
the
the
sunny
is
is
    #empty line 

2) remove empty lines

awk 'NF>0 {print}' words.txt |sort | uniq -c |sort -bnr

result

1 the sunny is is
1 the day is sunny the the

How do I merge above 2 scripts into 1 line of code to produce expected result?

4
  • Is it a requirement that you use awk? Or is a pipeline of other common Unix tools acceptable?
    – JigglyNaga
    Jul 24 '19 at 10:11
  • you need to insert a awk to filter line on length, this gave you an ugly awk -v RS=" " 'NF>0 {print}' words.txt | awk 'length' | sort | uniq -c |sort -bnr
    – Archemar
    Jul 24 '19 at 10:56
  • @JigglyNaga, it's actually from Leetcode question # 192: leetcode.com/problems/word-frequency you are free to use any commands
    – tchen003
    Jul 25 '19 at 22:16
  • @Archemar, great thanks it works! am trying to understand the relationship between | awk 'length' | and NF>0. Because without | awk 'length' | , system will still output blank lines despite NF>0. Could you share a bit on this?
    – tchen003
    Jul 25 '19 at 22:30
3

There is no real need to preprocess the text by changing spaces to newlines etc. especially not if you want to combine the operations into a single awk script.

$ awk '{ for (i=1; i<=NF; ++i) count[$i]++ } END { for (word in count) print count[word], word }' file
4 the
2 sunny
3 is
1 day

This walks over each whitespace-separated word and counts it. The counts are stored in the array count which is indexed by the words. At the end, the counts and the corresponding words are outputted. This would skip empty lines automatically, as they contain no words.

If you need this sorted, pipe it through sort -n.

If you use GNU awk, you can use its asorti() function to do the sort in the END block:

END {
    n = asorti(count, c2, "@val_num_desc")
    for (i=1; i<=n; i++ )
        printf("%d %s %s\n", i, count[c2[i]], c2[i])
}
4
  • @Archemar, thanks for the edit, but we don't really know what awk the user is using. I modified the text that you added.
    – Kusalananda
    Jul 24 '19 at 10:55
  • I've always found asort() and asorti() kinda clunky, you could just set PROCINFO["sorted_in"] if you're using gawk (newer versions).
    – Ed Morton
    Jul 24 '19 at 21:30
  • 1
    @EdMorton I've personally never used either. That piece of the answer was inserted by Archemar. I would sort outside of awk (my default awk is not GNU awk).
    – Kusalananda
    Jul 24 '19 at 21:31
  • In your first script instead of piping to sort -n you could just change END{ for to END{ PROCINFO["sorted_in"]="@val_num_desc"; for and leave the rest alone. It's an extremely useful feature (but not a big deal in this case).
    – Ed Morton
    Jul 24 '19 at 21:35
0

@Kusalananda already provided a good awk solution but alternatively:

$ tr ' ' '\n' < file | sort | uniq -c
      1 day
      3 is
      2 sunny
      4 the
0

If you have GNU grep, you can use the -o (--only-matching) option to display one match per line:

grep -o '\S\+' words.txt

Then pipe to sort and uniq as before.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.