0

I have a file that looks something like this:

A 1 abc
A 1 def
A 2 ttt
B 2 ppp
B 2 qqq

I'd like to keep the first line when the first two keys are identical, so something like this:

A 1 abc
A 2 ttt
B 2 ppp

I found a previous question that solved this. However, my file is 1.2 GB, and I'm matching on the first 19 columns. So when I run this:

  awk  '!array[$1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19]++' infile > outfile

I get this:

cmd. line:2: (FILENAME=infile FNR=287807) fatal: dupnode: r->stptr: can't allocate memory (Not enough memory)

Obviously I can't break the file apart to process because I don't know where the duplicates are. I'm willing to trade speed for memory to solve this. (The file is about 1.6 million lines).

3

As your file seems to be sorted:

sort -m -u -k 1,2 < file

-m for merge doesn't attempt to sort the file, but with -u (for unique) combined with -k 1,2 to specify a sort key consisting of the first 2 fields (use -k 1,19 for the first 19 fields), we're removing duplicates on the first 2 fields.

If the file isn't sorted (at least on those 2 fields), then just drop the -m. The result will end up being sorted though. The sorting will be expensive, but it should be OK memory wise as sort resorts to using temporary files for sorting big files (you need free disk space on /tmp (or $TMPDIR)).

| improve this answer | |
  • For my purposes, the file isn't really sorted. sort seems to complain after 2 fields; when I do -k 1,2,3,4 (all the way to 19), it gives me invalid field specification (in fact, it complains at 1,2,3) – innocentunremarkable Jan 28 '16 at 16:59
  • @innocentunremarkable, it's -k1,19 for fields 1 to 19 (for the portion of the line that starts at the first field and ends with the 19th field) – Stéphane Chazelas Jan 28 '16 at 17:01
0

Will combining the columns that are identical to form an index work ? With your above eg. we could do --

 awk '{ind=""; for(i=1; i<3; i++) {ind=ind" "$i }  if (!arr[ind]) arr[ind]=$3  } END{for (i in arr) print i, arr[i]}'

  A 1 abc
  A 2 ttt
  B 2 ppp

you will of course need to change the loop above to combine the indices you need.

| improve this answer | |
0

For future occasions, you can do so.. and you save the problem of giant files with arrays. Each time the field one changes its value, the array is deleted.

BEGIN{
    xd=""; 
}
{
    id=$1;

    if (id != xd)
    {
        for (x in arr)
        {
            print x,arr[x];
        }

        delete arr; #Each time the field one changes its value
        xd=id;
    }
    ind="";
    for (i=1; i<3; i++)
    {
        ind=ind $i;
    }
    if (!arr[ind])
    {
        arr[ind]=$3;
    }

}

END {
        for (x in arr)
        {
            print x,arr[x];
        }
}

Output:

$ awk -f script.awk file.txt
A1 abc
A2 ttt
B2 ppp
| improve this answer | |
0

Try this out:

awk '_a[$1" "$2]++==0'  < filename
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.