GNU AWK
Here's a test run with slightly modified input file:
bash-4.3$ $ awk '{v=sprintf("%s_%s",$1,$NF); if ( ! a[v]) print; a[v]++; }' input.txt
bash: $: command not found
bash-4.3$ awk '{v=sprintf("%s_%s",$1,$NF); if ( ! a[v]) print; a[v]++; }' input.txt
AAAAA stuff in between BBBBB
AA stuff AAABBBBB
AAAAA more stuff in the middle CCCCCC
bash-4.3$ cat input.txt
AAAAA stuff in between BBBBB
AA stuff AAABBBBB
AAAAA more stuff in the middle CCCCCC
AAAAA even more cool stuff BBBBB
AAAAA extra line CCCCCC
How this works:
- Key point: we build up an array of key-value pairs, and print based on whether we've already have an array item for that key-value pair
- key is constructed via
v=sprintf("%s_%s",$1,$NF)
where,$NF
is the last field in line; for example, for lines 1 and 3, the key would beAAAAABBBBB
. Value is an integer, incremented each time we've a match. v
variable and corresponding checkif (!a[v]) print
are performed on each line, printing is done only if value is not found in array.a[v]++
is performed on each line regardless of whether we printed or not
As Sundeep pointed out in the comments, this same approach can be simplified to :
bash-4.3$ awk '!seen[$1"_"$NF]++' input.txt
AAAAA stuff in between BBBBB
AA stuff AAABBBBB
AAAAA more stuff in the middle CCCCCC