I have a pipe-delimited file which can have millions of records. What I want to do is sort the file and remove if there are any duplicate lines there.
Then I need to check the first 32 columns and see if the content from first line matches with second line. If yes, then delete the second line and again check the first line with third line(which now will become second line due to removal of previous line since it was same). if this is also same then remove the third line as well and continue to compare first line with subsequent lines from file until we get a mismatch. For example: Input File:
a|a1|a2|a3|a4|...|firstname.lastname@example.org a|a1|a2|a3|a4|...|email@example.com$1553:2015-02-14 a|a1|a2|a3|a4|...|firstname.lastname@example.org:2015-03-01 a|a1|a2|a3|a4|...|email@example.com$121:2015-01-31 a|a1|a2|a3|a4|...|firstname.lastname@example.org$293:2015-02-28 b|b1|b2|b3|b4|...|email@example.com$542:2013:05:24 b|b1|b2|b3|b4|...|firstname.lastname@example.org$542:2013:05:24 b|b1|b2|b3|b4|...|email@example.com:2013:05:24 b|b1|b2|b3|b4|...|firstname.lastname@example.org$542:2013:05:24 b|b1|b2|b3|b4|...|email@example.com:2014:05:24 c|c1|c2|c3|c4|...|c32|Asce@ita.in c|c1|c2|c3|c4|...|c32|$200:2011:12:06 c|c1|c2|c3|c4|...|firstname.lastname@example.org$214:2001:01:31
There are 33 fields in above example and "..." is used to represent the continuity. So here as you see, first 4 lines are same (comparing only till field number 32 which is shown in example having a value of "a32"). So I need to keep just the 1st one and delete others. Similarly for columns starting with
b, we have 5 records which are the same, so again delete the last 4 and keep the first one.
For records starting with
c, we just have 3 records so we need to delete last 2 and keep 1st one.
Hence the output file would be some thing like this:
Required Output file:
a|a1|a2|a3|a4|...|email@example.com b|b1|b2|b3|b4|...|firstname.lastname@example.org$542:2013:05:24 c|c1|c2|c3|c4|...|c32|Asce@ita.in
I was thinking of using a regular loop and sed command to get this done but it slows the process significantly.
So, is there a way where we can use
ack and make this search/manipulation faster?