Return to Revisions

GNU AWK

Here's a test run with slightly modified input file:

bash-4.3$ $ awk '{v=sprintf("%s_%s",$1,$NF); if ( ! a[v]) print;   a[v]++;   }' input.txt 
bash: $: command not found
bash-4.3$ awk '{v=sprintf("%s_%s",$1,$NF); if ( ! a[v]) print;   a[v]++;   }' input.txt 
AAAAA stuff in between BBBBB
AA stuff AAABBBBB
AAAAA more stuff in the middle CCCCCC

bash-4.3$ cat input.txt
AAAAA stuff in between BBBBB
AA stuff AAABBBBB
AAAAA more stuff in the middle CCCCCC
AAAAA even more cool stuff BBBBB
AAAAA extra line CCCCCC

How this works:

• Key point: we build up an array of key-value pairs, and print based on whether we've already have an array item for that key-value pair
• key is constructed via v=sprintf("%s_%s",$1,$NF) where, $NF is the last field in line; for example, for lines 1 and 3, the key would be AAAAABBBBB. Value is an integer, incremented each time we've a match.
• v variable and corresponding check if (!a[v]) print are performed on each line, printing is done only if value is not found in array.
• a[v]++ is performed on each line regardless of whether we printed or not

As Sundeep pointed out in the comments, this same approach can be simplified to :

bash-4.3$ awk '!seen[$1"_"$NF]++' input.txt
AAAAA stuff in between BBBBB
AA stuff AAABBBBB
AAAAA more stuff in the middle CCCCCC