I would appreciate your help. I am extracting information from data files that are in some cases more than a terabyte in size.

  • Variables on each line are are separated by whitespace.
  • Quantity of variables on each line is fixed for each file
  • Right three columns always natural numbers
  • Lines always begin with an array
  • Arrays always contain fixed quantity of elements per file
  • Arrays can contain 1 to 5 elements
  • Source data file is properly sorted

Sample below compares a three element array to every other array in the file or chunk when using parallel. If the array matches, the second column from the right is added, and the lines are merged. Right most column and -2 column are flushed.

g@grml # zcat googlebooks-eng-us-all-3gram-20120701-zz.gz | head
Z'Z . _END_     1840    1       1
Z'Z . _END_     1847    1       1
Z'Z . _END_     1850    1       1
Z'Z . _END_     1855    1       1
Z'Z . _END_     1856    1       1
Z'Z . _END_     1857    1       1
Z'Z . _END_     1860    1       1
Z'Z . _END_     1863    1       1
Z'Z . _END_     1865    1       1
Z'Z . _END_     1869    1       1

g@grml # zcat googlebooks-eng-us-all-3gram-20120701-zz.gz | parallel -k -q --pipe awk '{a[$1" "$2" "$3] +=$(NF-1)} END{for (i in a) print i, a[i]}' | head 
Zz_NOUN _NOUN_ , 98
zz _._ _PRT_ 120
ZZ or_CONJ _NOUN_ 122
zz_DET _NOUN_ . 86
ZZ is_VERB reached 42
ZZ_NUM ^ ^ 65
ZZ _NOUN_ _VERB_ 3163
ZZ ,_. " 52
ZZ / _NUM_ 275

Although the example depicts a 3 element array, I am working with arrays containing 1 to 5 elements.

awk '{a[$1] +=$(NF-1)} END{for (i in a) print i, a[i]}'
awk '{a[$1" "$2] +=$(NF-1)} END{for (i in a) print i, a[i]}'
awk '{a[$1" "$2" "$3] +=$(NF-1)} END{for (i in a) print i, a[i]}'
awk '{a[$1" "$2" "$3" "$4] +=$(NF-1)} END{for (i in a) print i, a[i]}'
awk '{a[$1" "$2" "$3" "$3" "$5] +=$(NF-1)} END{for (i in a) print i, a[i]}'

How do I tell awk to compare the current array to the previous array on the previous line instead of having awk attempt to match every array to every array in a file or chunk?

Thank you

Example source file.

wget --show-progress -cq http://storage.googleapis.com/books/ngrams/books/googlebooks-eng-us-all-3gram-20120701-zz.gz -O - | zcat
  • So the number fields are not separated by tabs from the first data (the "array") on each line? – Kusalananda Jun 10 at 16:10
  • cant understand the problem. please provide sample output to given input. – Dudi Boy Jun 11 at 20:41

The input data given as an URL in a comment is tab delimited. This means we can parse its first tab-delimited field as a sort of "key" to compare between with other lines. We do not have to care about the space separated words within the first field, but can treat the whole first field as a single entity.

BEGIN { OFS = FS = "\t" }

    count = $(NF - 1)
    key = $1

key != previous {
    if (previous != "")
        print previous, sum

    sum = 0

    sum += count
    previous = key

    if (previous != "")
        print previous, sum

This awk program parses out the "count" field (second to last field) into count, then uses the first field as the "key" to later compare with the key of the previous line. This is the first block after the BEGIN block (which just sets input and output delimiters).

If the key is different from the key on the previous line, this means we're now looking at some other set of words. Output the previous line's key and sum, and reset the sum.

For all lines, increment the sum by the count from this line, and update previous (we're now done with this line, so this line's key is the next one's previous).

At the end, output the info for the last line in the data.

You would run this using awk -f script.awk inputfile.

As a "one-liner":

awk -F '\t' 'BEGIN{OFS=FS} {c=$(NF-1);k=$1} k!=p {if(p!="")print p,s;s=0} {s+=c;p=k} END {if(p!="") print p,s}' file
  • Thank you, it appears to work well. I apologize for the confusion about the pasting. – J363 Jun 10 at 17:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.