1

I have two files:

Ref.txt which contains:

ABCDEFG,15147,ABC,ABCD,ABCDE
HIJKLMN,500,HIJKLM,HIJKL,HIJK

and Seq.txt which contains:

ABCDEFG
ABCXXXX
ABCDXXX
ABCDEXX
HIJKXXX
HIJKLXX
HIJKLMX
HIJKLMN

I would like to search for fields 3-end of each line of the Ref.txt file in the Seq.txt file and replace the lines where that string is found with the string in the first field of that line. The output I am lookin for in this example is this:

ABCDEFG
ABCDEFG
ABCDEFG
ABCDEFG
HIJKLMN
HIJKLMN
HIJKLMN
HIJKLMN

Although this is incorrect code I am thinking of a command like this:

awk '{for(i=3; i<=NF; i++) gsub( $i , $1)}

but where the for loop refers to the the Ref.txt file and the gsub command is executed on the Seq.txt file.

2

One way you might consider doing this is to construct a single regular expression from fields 3-end of each line in Ref.txt by chaining them with |, saving that in associative array keyed on the $1 value.

You can then loop over the keys for each $1 of Seq.txt:

awk -F, '
  NR==FNR {
    # construct a single ERE as $3|$4|$5|etc. keyed on $1
    for(i=3;i<=NF;i++) r[$1] = r[$1] == "" ? $i : r[$1] "|" $i;
    next
  } 
  {
    # test $1 against each ERE and substitute the first matching key
    for(k in r) {
      if ($1 ~ r[k]) {
        $1 = k; 
        break
      }
    }
  }
  1
' Ref.txt Seq.txt
  • this seems to be working great so far! Thank you! I think I generally understand how this is working, but I am very new to coding, if you have time I would appreciate a detailed explanation of what is being done in each part of this script so I can modify it myself in the future if I encounter any problems. – DNA_bash Aug 14 at 12:07
  • How would you change the second half of the script if instead of the array being filled with sequences to search for it is filled with line numbers, and I would line to change those lines to the content of the first field of the Ref.txt file? Basically get the same output but using line numbers instead of searching for a sequence. – DNA_bash Aug 16 at 16:40
  • 1
    @DNA_bash so $3 to $NF would be a list of line numbers, rather than a list of patterns? if so, the above approach would not really be suitable – steeldriver Aug 16 at 17:31
  • that is correct, since a different approach is needed for that problem I have posted a new question to deal with using line numbers as an input here: unix.stackexchange.com/questions/535927/… – DNA_bash Aug 16 at 18:49
0

You can accomplish this with sed using just the Posix constructs as shown here:

$ sed -ne '
   /,/!G
   y/\n_/_\n/
   s/^\([^_][^_]*\).*__\(.*_\)\{0,1\}\([^,]*\),[^,]*,[^_]*,\1,.*/\3/p
   s/^[^_]*__//;s/$/,/
   y/_\n/\n_/
   /\n/!H
' Ref.txt Seq.txt

Note: this is run on a bash command line and the order and number of arguments is as given here.

Perl can also be used here without taking recourse of the regexes:

$ perl -F, -lane '
   if ( @ARGV ) {
     $h{$_} = $F[0] for @F[2..$#F];
     next;
   }
   my $seq = $_;
   my($k) = grep { ! index($seq, $_) } keys %h;
   print $h{$k};
' Ref.txt Seq.txt
-1

If you are lazy like me, you don't want to deal with arrays and loops and leave that up to your tool. That's why I prefer sed and just throw those Ref.txt lines (identified by the comma) on the hold space with H. Actually I add another comma, so I know that patterns are always in between two commas. So together it's /,/{s/$/,/;H;d;}.

Now, for each line of Seq.txt, I append the hold space with my reference pile with G and I let the s command replace the pattern with the pattern found in the pile. As often, the regular expression is easier to write than to read.

Let's look at the pattern space now:

ABCDEFG\n\nABCDEFG,15147,ABC,ABCD,ABCDE\nHIJKLMN,500,HIJKLM,HIJKL,HIJK
\_____/    \_____/       \_/
replace    by this  if  match
  • The part to be replaced is at the beginning (^) with a matching part ([A-Z]+) and some parts before and after ([A-Z]*)
  • The substitute is a sequence between a newline and a comma: \n[A-Z]*,
  • the match is a pattern between two comma, backreferenced from the first part: ,\1,

All together this gives

sed -E '/,/{s/$/,/;H;d;};G;s/^[A-Z]*([A-Z]{1,})[A-Z]*\n.*\n([A-Z]*),[A-Z0-9,]*,\1,.*/\2/;P;d' Ref.txt Seq.txt
  • hmm, this solution doesn't appear to work in my hands. I am very new to coding in general so could be my implementation, but running this outputs a copy of the Seq.txt file with no changes – DNA_bash Aug 14 at 12:04
  • I only could test it with GNU sed, and I see it doesn't work with option --posix. Currently I don't see why, but replacing the [A-Z]+ with [A-Z]{1,} seems to help, although those should behave the same. Sorry for that. Edited the answer. – Philippos Aug 14 at 12:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.