-1

I think it's easier if I break this question into parts.

  1. Is there a way to find lines that have the same values in the first field and the same values in the last field? - but not necessarily equal to each other?

Lines:

  AAAAA stuff in between BBBBB
  AAAAA more stuff in the middle CCCCCC
  AAAAA even more cool stuff BBBBB

Lines of interest:

  AAAAA stuff in between BBBBB
  AAAAA even more cool stuff BBBBB
  1. Is there a way to remove all but one of the lines that have the same first and last fields duplicated?

Before:

  AAAAA stuff in between BBBBB
  AAAAA more stuff in the middle CCCCCC
  AAAAA even more cool stuff BBBBB

After:

  AAAAA stuff in between BBBBB
  AAAAA more stuff in the middle CCCCCC

I can process the file a little bit in order to simplify the solution - say set a fixed number of fields.

Currently, the file does not have the same number of fields in each line, but the first and last fields are of interest.

Spaces are between all the fields.

I have searched and have found some references to awk but I could not find examples of awk using multiple fields in this way.

Big thanks.

marked as duplicate by Sundeep, Archemar, Stephen Rauch, Anthon, Romeo Ninov Jul 5 '17 at 7:20

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • Is the after supposed to be AAAAA stuff in between BBBBB AAAAA even more cool stuff BBBBB – Modelmat Jul 5 '17 at 5:50
0

GNU AWK

Here's a test run with slightly modified input file:

bash-4.3$ $ awk '{v=sprintf("%s_%s",$1,$NF); if ( ! a[v]) print;   a[v]++;   }' input.txt 
bash: $: command not found
bash-4.3$ awk '{v=sprintf("%s_%s",$1,$NF); if ( ! a[v]) print;   a[v]++;   }' input.txt 
AAAAA stuff in between BBBBB
AA stuff AAABBBBB
AAAAA more stuff in the middle CCCCCC

bash-4.3$ cat input.txt
AAAAA stuff in between BBBBB
AA stuff AAABBBBB
AAAAA more stuff in the middle CCCCCC
AAAAA even more cool stuff BBBBB
AAAAA extra line CCCCCC

How this works:

  • Key point: we build up an array of key-value pairs, and print based on whether we've already have an array item for that key-value pair
  • key is constructed via v=sprintf("%s_%s",$1,$NF) where, $NF is the last field in line; for example, for lines 1 and 3, the key would be AAAAABBBBB. Value is an integer, incremented each time we've a match.
  • v variable and corresponding check if (!a[v]) print are performed on each line, printing is done only if value is not found in array.
  • a[v]++ is performed on each line regardless of whether we printed or not

As Sundeep pointed out in the comments, this same approach can be simplified to :

bash-4.3$ awk '!seen[$1"_"$NF]++' input.txt
AAAAA stuff in between BBBBB
AA stuff AAABBBBB
AAAAA more stuff in the middle CCCCCC
  • can be simplified to awk '!seen[$1,$NF]++' input.txt – Sundeep Jul 5 '17 at 5:38
  • 1
    and your solution will fail if you add AAA stuff in between AABBBBB to input file because you are only concatenating first and last field – Sundeep Jul 5 '17 at 5:41
  • @Sundeep Good point. That can be worked around by adding a character in between within sprintf. I'll edit that in a minute – Sergiy Kolodyazhnyy Jul 5 '17 at 5:44

Not the answer you're looking for? Browse other questions tagged or ask your own question.