0

I am given a txt file (war and peace..), and i need to create a text file sorted alphabetically of all the words that appear 10 or more times (without the quantity).

The twist in this question, is that every punctuation is considered as a beginning of a new word, meaning you're is considered two words, you re.

I flipped all punctuation into new lines, and all spaces to new lines. And i used trim -c so now i have all words and their count, don't know how to show only those that appear 10 or more times.

Any help regarding a way to find all words that appear 10 or more times would be really appreciated!

  • 3
    Solve your homework yourself! – Ipor Sircer Nov 29 '16 at 14:45
  • I flipped all punct for new lines, and all spaces to new lines. trimmed it (so now i have all words in the text, and i know how much each one appears. What I don't know is from this state how to only show those that appear 10 or more times – Michael Segal Nov 29 '16 at 14:51
3
< text tr -cs '[:alnum:]' '[\n*]' |
  awk '++count[$0] == 10' |
  sort

Replace $0 with tolower($0) if you want to ignore case.

That translates sequences of characters that are the complement of the alphanumerical ones to newlines. awk prints the 10th occurrence of each.

Note that on GNU systems, tr doesn't work properly on multi-byte characters. However, on those systems, you can use GNU grep's -o extension instead:

< text grep -Eo '[[:alnum:]]+' |
  awk '++count[$0] == 10' |
  sort

You can change that to

< text grep -Eo '[^[:punct:][:space:]]+' |
  awk '++count[$0] == 10' |
  sort

to consider characters that are neither punctuation nor space (or tr -s '[:punct:][:space:]' '[\n*]' above for non-GNU system or all-ASCII text) which on that War and Peace text gives the same result.

Note that on GNU systems at least, that could still give wrong results as Unicode combining accents for instance are classified as punctuation and not alnums (they don't appear in that text though where the accented characters are in their combined form).

1

In perl :

perl -e 'while(<>){while(/([[:alpha:]]+)/g){$c{$&}++}};foreach $w(keys %c){if($c{$w}<10){delete $c{$w}}};foreach $w(sort keys %c){print "$w\n"}' WarAndPeace.txt

Indented :

perl -e '# Let s count every words
         while(<>){
             while(/([[:alpha:]]+)/g){
                 $c{$&}++
             }
          }
          # Let s remove lesser than 10
          foreach $w(keys %c){
              if($c{$w}<10){
                  delete $c{$w}
              }
          }
          # Let s sort and print
          foreach $w(sort keys %c){
              print "$w\n"
          }' WarAndPeace.txt

With UTF-8 :

 perl -e 'use open qw/:std :utf8/;
          # Let s count every words
             while(<>){
                 while(/([[:alpha:]]+)/g){
                     $c{$&}++
                 }
              }
              # Let s remove lesser than 10
              foreach $w(keys %c){
                  if($c{$w}<10){
                      delete $c{$w}
                  }
              }
              # Let s sort and print
              foreach $w(sort keys %c){
                  print "$w\n"
              }' WarAndPeace.txt
  • I just added the needed command for utf-8 – Vouze Nov 29 '16 at 16:16
1

In case you have already stripped dots and white-spaces from the file and only have the words in it, as mentioned in your comments (no extra white-spaces/tabs), then you can use sort,uniq and awk to get what you wanted :

sort file_with_words | uniq -c | awk '{ if ($1 >= 10) { print}}'

My solution only works if the file is properly formatted (one word / line, no extra spaces/tabs).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.