2

I want to remove duplicate/inverse match pairs from a text file.

E.g., the file contains data like the below:

10 |Name1 |20 |Name2
20 |Name2 |30 |Name3
20 |Name2 |10 |Name1   <-- Inverse pair (compared to first line) to be removed from text file
40 |Name4 |30 |Name3

I am expecting output like the below:

10 |Name1 |20 |Name2
20 |Name2 |30 |Name3
40 |Name4 |30 |Name3
3

with awk:

awk -F '[[:blank:]]*[|][[:blank:]]*' -v SUBSEP='|' '
    ($1,$2,$3,$4) in seen || ($3,$4,$1,$2) in seen {next} 
    {seen[$1,$2,$3,$4]; print}
' file

That sets the field separator to the pipe character with optional leading or trailing whitespace. Then it looks for the "pairs" as associative array keys, in either order: if found, skip this line; otherwise, add the key to the array and print the line.

1

Maybe something like:

perl -F'[|]' -lane '
  for (@F) {
    # trim the fields to remove leading and trailing blanks
    s/^\s+//; s/\s+$//
  }
  # re-join the trimmed fields into $a
  my $a = join "|", @F[0..3];

  # same, inverting the two pairs into $b
  my $b = join "|", @F[2,3,0,1];

  # print unless either $a or $b has been seen before
  print unless $seen{$a} || $seen{$b}++' < your-file

To generalise to any number of pairs found in any order, you'd want to sort those pairs to make the key of the %seen associative array:

perl -F'[|]' -lane '
  for (@F) {
    # trim the fields to remove leading and trailing blanks
    s/^\s+//; s/\s+$//
  }

  my @pairs;
  while (my ($a, $b) = splice(@F, 0, 2)) {
    push @pairs, "$a|$b"
  }
  my $key = join "|", sort @pairs;

  print unless $seen{$key}++' < your-file
0

You can do this with sed. The code below is using gnu sed, but it can readily be made posix compliant.

sed -Ee '
   $!{
      s/$/|/
      N
      s/[[:blank:]]+//g
      H;s/.*//;x;D
   }
   s/$/|/
   G;H;g
   y/\n_/_\n/
   :xdup
       s/_((([^_|]+[|]){2})(([^_|]+[|]){2}))_(.*_)?\4\2_/_\1_\6/
   txdup
   s/^_//;s/_$//
   y/\n_/_\n/
'  input.csv

Will explain later as rush hour approacht

Results :

10|Name1|20|Name2|
20|Name2|30|Name3|
40|Name4|30|Name3| 

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.