0

I have 10 text files, in each file i have a chapter from a book, i want to find the pair of words that appear the most of the time together in a line i.e:

chapter1:

hello world good boy green sun

good green boy sun world hello

chapter2:

chapter3:

.....etc

Output wanted for chapter1:

hello world (alphabet order)
  • 3
    The output should have "boy green" also, right? – Guru Apr 17 at 14:45
  • yes "boy green" also, didn't see – John B Apr 17 at 15:00
  • 1
    what about if you had a line like Hello world Hello? it should come in output? and how ? Hello world or world Hello? – αғsнιη Apr 17 at 15:02
  • hello world , and this will count for 2 to the pair "hello world" always alphabetical – John B Apr 17 at 15:10
  • 1
    how about Hello\nworld\nHello? \n is actual new-line character. please edit your question to answer for comments asking for clarifications – αғsнιη Apr 17 at 15:31
2
awk '
  {
    $0 = tolower($0)
    for (i = 1; i < NF; i++) {
      pair = $i"" < $(i+1) ? $i" "$(i+1) : $(i+1)" "$i
      c = ++count[pair]
      if (c > max) max = c
    }
  }
  END {
    for (pair in count)
      if (count[pair] == max)
        print pair
  }'
0

Try this,

  1. Use awk to print each pair of words.
  2. Use perl to sort the words in a pair (via).
  3. Use sort and uniq -c to count occurrences each pair.

awk '{for (i=1;i<NF;i++) { print tolower($i)" "tolower($(i+1)) }}' file \
| perl -ane '$,=" "; print sort @F; print "\n";' \
| sort | uniq -c | sort -b -k1nr -k2

Output:

  2 boy green
  2 hello world
  1 boy good
  1 boy sun
  1 good green
  1 good world
  1 green sun
  1 sun world
  • can not use perl or pipeline.. – John B Apr 17 at 15:35
  • 2
    Why can't you, @John? Those are standard utilities on most Linux systems. – Jeff Schaller Apr 17 at 15:37
  • 1
    Yes, this kind of information should be in your question. – pLumo Apr 17 at 15:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.