text processing (reading and computing from two files) [awk, script]

Question

I have one text like the following: (words, word domains)

car transport
car machine
bank economy
bank politics
bank parks
God religion
...

There are numerous words, some words have different domains and other have only one domain. I have another file, a huge matrix (300 dimensions each line) composed of words and a vector for each like this:

bank 0.9 1.5 3.2 -0.2 0.1 ... 
God 1.0 2.1 -0.5 0.7 ...
rose 0.2 -1.8 ...
... ...

I would like to read how many times appears each word in the first file, and according to that choose the highest "n" numbers in each vector of the second file, knowing to which field it belongs. Something like this:

car 2
bank 3
God 1

and pass those numbers to

bank 4 3.2
bank 3 1.5
bank 2 0.9
God 3 2.1

The first part I thought of

gawk 'NR==FNR {a[$1]++;next;} dont know what here?' list matrix

I know its a little bit complex, but any help is appreciated. Maybe another way is easier?

Answer 1

awk '
    NR==FNR{                                #operate matrix file first
        A[$1] = 1                           #array of words
        for(i=2;i<=NF;i++)
            B[$1 OFS i] = $i                #array with indexes [word field_num]
        next
        }
    $1 in A{                                #if word in array A
        max = $1 OFS 2
        for(i in B)
            if(i ~ "^" $1 && B[max] < B[i])
                max = i                     #find maximum in B-array
        print max, B[max]                   #output word + field_num + value
        delete B[max]                       #exclude value from next search 
        }
    }
    ' matrix list

If awk version allows pseudo multidimention arrays script can be simplified

awk '
    NR==FNR{                                
        for(i=2;i<=NF;i++)
            A[$1][i] = $i                   
        next
        }
    $1 in A{
        max = 2
        for(i in A[$1])
            if(A[$1][max] < A[$1][i])
                max = i
        print $1, max, A[$1][max]
        delete A[$1][max]
        }
    }
    ' matrix list

Answer 2

That is pretty complicated indeed. I would suggest creating a awk script, unless someone comes up with a miracle one-liner.

Inside your awk file:

NR==FNR {

    a[$1]++
    next

} #Your probably know what that does since it's your starting point

# If first field is a key in array a
$1 in a { 
    # Assign the number of occurences of this word in variable n
    n=a[$1]  
    # Initialize this value to + INFINITY  
    k=-log(0)

    # Loop on the number of occurences of the word
    for (i=0; i<n; i++) {
        # Initialize max value and its index at the first value of the vector
        m=$2
        i_m=2

        # Loop on the number of fields in the matrix for that word
        for (j=3; j<NF+1; j++) {

            # Look for the largest value that stays below previous max (if none then k is INFINITY)
            if ($j > m && $j < k) { m=$j; i_m=j }

        }
        # Print the word, the index of its max and its value
        printf $1" "i_m" "m"\n"
        # Store the max to be able to scan for the next biggest number at next iteration
        k=m
    }

}

To run it:

$ awk -f myScript.awk list matrix

My script seems to work okay except for one case: if there is an equal number or more occurences of a word in list than there are values in its vector in matrix. That doesn't seem to be the problem here since your vectors are pretty big. Also the initialization of k at -log(0) to get it to inf value is a bit weird but I couldn't figure out how to set it to inf directly (=inf does not work obviously). You can probably make it handle more cases (if you have the same value several times in your vector for example...) but I'll leave it to you as you have a starting point now !

Answer 3

TXR Lisp with awk macro:

(let ((h (hash :equal-based)))
  (awk (:inputs "word-dom-pairs")
    (t (inc [h [f 0] 0])))
  (awk (:inputs "word-vectors")
    (t (whenlet ((count [h [f 0]]))
         (fconv - : r)
         (let* ((n-fn-pairs (zip (rest f) (range 2)))
                (n-fn-sorted [sort n-fn-pairs > first]))
           (each ((p [n-fn-sorted 0..count]))
             (prn [f 0] (second p) (first p))))))))

Run:

$ txr munge.tl 
bank 4 3.2
bank 3 1.5
bank 2 0.9
God 3 2.1

Data:

$ cat word-dom-pairs 
car transport
car machine
bank economy
bank politics
bank parks
God religion

$ cat word-vectors 
bank 0.9 1.5 3.2 -0.2 0.1
God 1.0 2.1 -0.5 0.7
rose 0.2 -1.8

Here is a verison of the program rolled into a single awk expression:

(awk (:inputs "word-dom-pairs" "word-vectors")
     (:let (h (hash :equal-based)))
     ((= arg 1) (inc [h [f 0] 0]))
     ((= arg 2) (whenlet ((count [h [f 0]]))
                  (fconv - : r)
                  (let* ((n-fn-pairs (zip (rest f) (range 2)))
                         (n-fn-sorted [sort n-fn-pairs > first]))
                    (each ((p [n-fn-sorted 0..count]))
                      (prn [f 0] (second p) (first p)))))))

The two :inputs from the previously separate awk-s are merged into one. We replace the unconditionally true patterns t with selectors based on which input is being processed given by the arg variable. The let which binds the hash table variable is folded into a the awk macro :let clause.

If we remove the (:inputs ...) clause, we can give the files using a pair of command line arguments:

$ txr munge.tl file1 file2

TXR Lisp is a type-safe, dynamic language in which variables have to be defined before assignment or use. Nonexistent variables and junk strings aren't numeric zero, and strings which look like numbers aren't those numbers. This is why we define the hash table's existence explicitly, and use fconv to explicitly convert the second and subsequent fields to real numbers (r).