1

I have two data files:

  1. file_1.in, containing over 2k lines like "12 AB0001":

    10 AB0001
    11 AC0002
    12 AD0003
    ...
    
  2. A list of *.gz gzipped files (about 1 to 3 million lines) I should extract and parse to create one output file named as lines (second col) in file_1.in.

    ##comment..
    ##comment..
    #CHROM POS ID REF ALT QUAL FILTER INFO FORMAT AB0001 AC0002 AD0003
    21 1234567 ab11111 G A 100 PASS info1;info2 GT 0|0 0|1 0|0
    21 1234568 ab22222 C A 100 PASS info1,info2 GT 1:23:2 .:.:. 0:32:2
    21 1234569 ab33333 A C 100 PASS info1;info2 GT 0|2 1|0 0|0
    

Trying different approaches I came to this:

{
if(FNR==NR){list[$1]=$2;next}
if(!/^#/){
    for(p in list){
        split($p, sp, ":");
        if (sp[1] != "0|0" && sp[1] != "0" && sp[1] != "."){
            printf("%s %s %s %s %s %s %s %s %s %s\n", 
            $1, $2, $3, $4, $5, $6, $7, $8, $9, $p) >> out"/"list[p]".tmp"
        }
    }
} 
}

executed by command line:

awk -v out="outfolder/" -f myscript.awk file_1.in <(zcat *.gz)

But it takes more than two hours for creating just one file. There could be a way to improve my code? I think most of the time is spent by zcat each file and by the append writing function that seems to be slow. What do you think?

13
  • 3
    I’m missing the zcat portion...
    – Jeff Schaller
    Nov 19, 2017 at 0:55
  • Can you add some data, that would help me understand what you're doing :)
    – tink
    Nov 19, 2017 at 2:01
  • I still can't figure out what you expect the value of $p to be ...
    – tink
    Nov 19, 2017 at 3:11
  • @JeffSchaller sorry I forgot the command, just edited, it won't work without.
    – redrich
    Nov 19, 2017 at 10:12
  • @tink just added few examples with almost all cases. $p will be the index of the column I need that will be written to a single file named as that column. I know that it is possibile to process the column without taking file_1.in as input but I've noticed that doing a for loop on more then 2k columns in 24 files takes too much time, so I came up with this solution (not sure if is the best one) as the header of the files is always the same
    – redrich
    Nov 19, 2017 at 10:20

1 Answer 1

0

This code does a lot of processing in the loop that is the same between iterations. Awks do not have sophisticated loop optimization to be able to hoist that out of the loop:

    for(p in list){
        split($p, sp, ":");
        if (sp[1] != "0|0" && sp[1] != "0" && sp[1] != "."){

The list associative array does not change after the first file is read to populate it, yet you're repeatedly splitting the index values to test this condition. Instead you could make one pass through the array, and delete the entries which don't match. Or ... not insert those entries into the array in the first place while processing file_1.in! Then the splitting and test can be removed: the loop unconditionally visits every entry in list.

            printf("%s %s %s %s %s %s %s %s %s %s\n", 
            $1, $2, $3, $4, $5, $6, $7, $8, $9, $p) >> out"/"list[p]".tmp"

The material being printed here is the same for each iteration of the loop, except for the last %s field which takes $p. You could use sprintf outside of the loop to format the nine fields into a string str and then do this: printf("%s %s", str, $p) >> out "/" list[p] ".tmp".

Something which is unclear is the following: the $p expression in the printf assumes that p is an integer which is suitable for indexing into fields. So the entire split logic looks bogus: it's spltting p on the : character and then testing certain fields of that split. If p is expected to be a positive integer, it shouldn't contain any colons. Might it be that you intended to test the second column here, which is list[p]?

Other than that, the program may likely be implicitly slow. It maintains thousands of open file descriptors, through which it adds small amounts of material to numerous output files. Millions of lines are coming in from the compressed files: for each line, the loop iterates over an assoc array containing over two thousand entries, and generates output into that many different files. So millions of compressed lines become billions of uncompressed lines.

You may want to ask yourself what is the purpose of unfolding the data into that representation? If it is for some subsequent processing, maybe a more space-efficient representation could work. The awk code looks like it is massively precomputing an association which some kind of database may be able to handle implicitly.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .