2

I want to find keywords, which often occur in relation to each other.

Example

A directory contains markdown files, each with some keywords on the last line:

$ tail -n 1 file1.md
#doctor #donkey #plants

$ tail -n 1 file2.md
#doctor #firework #university

$ tail -n 1 file3.md
#doctor #donkey #linux #plants

Pseudo output

  • 100% of the files containing the keyword "#donkey" also contain the keyword "#doctor".
  • 50% of the files containing the keyword "#plants" also contain the keyword "#linux".

A Shell script, an awk script, or just an explanation on how to achieve this goal would suffice!

Any help would be appreciated. Thank you very much

2
  • 1
    Hi Luca! What you're asking for is a clustering algorithm; neither shells nor simple text-based processing tools like awk will get you very far here. May 26 at 20:05
  • you could just build a large (lower traingular) table of keywords-vs-keywords and increase each (row n, column m) entry if the word in the nth row appears with the word in the word in the m'th column, and normalize by the total number of occurences of each word. Then, look for maximum subdiagonal elements. May 26 at 20:13

1 Answer 1

7

Using GNU awk for arrays of arrays:

If the keywords were on the first line of each file than also using GNU awk for nextfile for efficiency:

$ cat tst.awk
FNR == 1 {
    for ( i=1; i<=NF; i++ ) {
        words[$i]++
        for ( j=i+1; j<=NF; j++ ) {
            pairs[$i][$j]++
            pairs[$j][$i]++
        }
    }
    nextfile
}
END {
    for ( word1 in pairs ) {
        for ( word2 in pairs[word1] ) {
            pct = pairs[word1][word2] * 100 / words[word1]
            printf "%d%% of the files containing the keyword \"%s\" also contain the keyword \"%s\".\n", pct, word1, word2
        }
    }
}

$ awk -f tst.awk file*.md
100% of the files containing the keyword "#university" also contain the keyword "#doctor".
100% of the files containing the keyword "#university" also contain the keyword "#firework".
100% of the files containing the keyword "#plants" also contain the keyword "#donkey".
50% of the files containing the keyword "#plants" also contain the keyword "#linux".
100% of the files containing the keyword "#plants" also contain the keyword "#doctor".
100% of the files containing the keyword "#donkey" also contain the keyword "#plants".
50% of the files containing the keyword "#donkey" also contain the keyword "#linux".
100% of the files containing the keyword "#donkey" also contain the keyword "#doctor".
100% of the files containing the keyword "#linux" also contain the keyword "#plants".
100% of the files containing the keyword "#linux" also contain the keyword "#donkey".
100% of the files containing the keyword "#linux" also contain the keyword "#doctor".
33% of the files containing the keyword "#doctor" also contain the keyword "#university".
66% of the files containing the keyword "#doctor" also contain the keyword "#plants".
66% of the files containing the keyword "#doctor" also contain the keyword "#donkey".
33% of the files containing the keyword "#doctor" also contain the keyword "#linux".
33% of the files containing the keyword "#doctor" also contain the keyword "#firework".
100% of the files containing the keyword "#firework" also contain the keyword "#university".
100% of the files containing the keyword "#firework" also contain the keyword "#doctor".

or on the last line then again relying on gawk for ENDFILE:

$ cat tst.awk
ENDFILE {
    for ( i=1; i<=NF; i++ ) {
        words[$i]++
        for ( j=i+1; j<=NF; j++ ) {
            pairs[$i][$j]++
            pairs[$j][$i]++
        }
    }
}
END {
    for ( word1 in pairs ) {
        for ( word2 in pairs[word1] ) {
            pct = pairs[word1][word2] * 100 / words[word1]
            printf "%d%% of the files containing the keyword \"%s\" also contain the keyword \"%s\".\n", pct, word1, word2
        }
    }
}

$ awk -f tst.awk file*.md

or still on the last line but more efficiently using tail+gawk:

$ cat tst.awk
{
    for ( i=1; i<=NF; i++ ) {
        words[$i]++
        for ( j=i+1; j<=NF; j++ ) {
            pairs[$i][$j]++
            pairs[$j][$i]++
        }
    }
}
END {
    for ( word1 in pairs ) {
        for ( word2 in pairs[word1] ) {
            pct = pairs[word1][word2] * 100 / words[word1]
            printf "%d%% of the files containing the keyword \"%s\" also contain the keyword \"%s\".\n", pct, word1, word2
        }
    }
}

$ tail -qn1 file*.md | awk -f tst.awk
4
  • 2
    even if you just did that guy's homework, I'm still amazed :) May 26 at 21:00
  • @MarcusMüller thanks! Every time I see a "this would be difficult with awk" type of comment I just can't help myself... :-).
    – Ed Morton
    May 26 at 21:03
  • 1
    ohhh, so I nerd-sniped you! May 26 at 21:04
  • 1
    Exactly. I tried but just couldn't click away. If anyone every posts a comment that you can't prove the Riemann Hypothesis with awk I'm screwed.
    – Ed Morton
    May 26 at 21:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.