Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange
2 added 179 characters in body
source | link

GNU AWK

Here's a test run with slightly modified input file:

bash-4.3$ $ awk '{v=sprintf("%s%s""%s_%s",$1,$NF); if ( ! a[v]) print;   a[v]++;   }' input.txt 
bash: $: command not found
bash-4.3$ awk '{v=sprintf("%s_%s",$1,$NF); if ( ! a[v]) print;   a[v]++;   }' input.txt 
AAAAA stuff in between BBBBB
AA stuff AAABBBBB
AAAAA more stuff in the middle CCCCCC

bash-4.3$ cat input.txt
AAAAA stuff in between BBBBB
AA stuff AAABBBBB
AAAAA more stuff in the middle CCCCCC
AAAAA even more cool stuff BBBBB
AAAAA extra line CCCCCC

How this works:

  • Key point: we build up an array of key-value pairs, and print based on whether we've already have an array item for that key-value pair
  • key is constructed via v=sprintf("%s%s""%s_%s",$1,$NF) where, $NF is the last field in line; for example, for lines 1 and 3, the key would be AAAAABBBBB. Value is an integer, incremented each time we've a match.
  • v variable and corresponding check if (!a[v]) print are performed on each line, printing is done only if value is not found in array.
  • a[v]++ is performed on each line regardless of whether we printed or not

As Sundeep pointed out in the comments, this same approach can be simplified to :

bash-4.3$ awk '!seen[$1"_"$NF]++' input.txt
AAAAA stuff in between BBBBB
AA stuff AAABBBBB
AAAAA more stuff in the middle CCCCCC

GNU AWK

Here's a test run with slightly modified input file:

bash-4.3$ awk '{v=sprintf("%s%s",$1,$NF); if ( ! a[v]) print;   a[v]++;   }' input.txt
AAAAA stuff in between BBBBB
AAAAA more stuff in the middle CCCCCC

bash-4.3$ cat input.txt
AAAAA stuff in between BBBBB
AAAAA more stuff in the middle CCCCCC
AAAAA even more cool stuff BBBBB
AAAAA extra line CCCCCC

How this works:

  • Key point: we build up an array of key-value pairs, and print based on whether we've already have an array item for that key-value pair
  • key is constructed via v=sprintf("%s%s",$1,$NF) where, $NF is the last field in line; for example, for lines 1 and 3, the key would be AAAAABBBBB. Value is an integer, incremented each time we've a match.
  • v variable and corresponding check if (!a[v]) print are performed on each line, printing is done only if value is not found in array.
  • a[v]++ is performed on each line regardless of whether we printed or not

GNU AWK

Here's a test run with slightly modified input file:

bash-4.3$ $ awk '{v=sprintf("%s_%s",$1,$NF); if ( ! a[v]) print;   a[v]++;   }' input.txt 
bash: $: command not found
bash-4.3$ awk '{v=sprintf("%s_%s",$1,$NF); if ( ! a[v]) print;   a[v]++;   }' input.txt 
AAAAA stuff in between BBBBB
AA stuff AAABBBBB
AAAAA more stuff in the middle CCCCCC

bash-4.3$ cat input.txt
AAAAA stuff in between BBBBB
AA stuff AAABBBBB
AAAAA more stuff in the middle CCCCCC
AAAAA even more cool stuff BBBBB
AAAAA extra line CCCCCC

How this works:

  • Key point: we build up an array of key-value pairs, and print based on whether we've already have an array item for that key-value pair
  • key is constructed via v=sprintf("%s_%s",$1,$NF) where, $NF is the last field in line; for example, for lines 1 and 3, the key would be AAAAABBBBB. Value is an integer, incremented each time we've a match.
  • v variable and corresponding check if (!a[v]) print are performed on each line, printing is done only if value is not found in array.
  • a[v]++ is performed on each line regardless of whether we printed or not

As Sundeep pointed out in the comments, this same approach can be simplified to :

bash-4.3$ awk '!seen[$1"_"$NF]++' input.txt
AAAAA stuff in between BBBBB
AA stuff AAABBBBB
AAAAA more stuff in the middle CCCCCC
1
source | link

GNU AWK

Here's a test run with slightly modified input file:

bash-4.3$ awk '{v=sprintf("%s%s",$1,$NF); if ( ! a[v]) print;   a[v]++;   }' input.txt
AAAAA stuff in between BBBBB
AAAAA more stuff in the middle CCCCCC

bash-4.3$ cat input.txt
AAAAA stuff in between BBBBB
AAAAA more stuff in the middle CCCCCC
AAAAA even more cool stuff BBBBB
AAAAA extra line CCCCCC

How this works:

  • Key point: we build up an array of key-value pairs, and print based on whether we've already have an array item for that key-value pair
  • key is constructed via v=sprintf("%s%s",$1,$NF) where, $NF is the last field in line; for example, for lines 1 and 3, the key would be AAAAABBBBB. Value is an integer, incremented each time we've a match.
  • v variable and corresponding check if (!a[v]) print are performed on each line, printing is done only if value is not found in array.
  • a[v]++ is performed on each line regardless of whether we printed or not