regex from two files

Question

I have two files that built of blocks of 4 lines, starting with a @. They are matching and in order. The files are too big (>10 GB) so I am putting here the first 3 blocks of each (12 lines).

1st: the R file (reference file):

@NB551168:120:HTKN2BGX5:1:11101:3598:1051 2:N:0:NATC
CAATCTCTAAAGTTT
+
AA/A/EEEE///EEE
@NB551168:120:HTKN2BGX5:1:11101:24202:1051 2:N:0:NTCG
TGAGACCGGGTGTTG
+
AAAAAAEEE///<AA
@NB551168:120:HTKN2BGX5:1:11101:4381:1051 2:N:0:NCTT
GCTACTCCTAAGGCA
+
A////6/////EE//

2nd: the I file (index file):

@NB551168:120:HTKN2BGX5:1:11101:3598:1051 1:N:0:NATC
NATC
+
#EEE
@NB551168:120:HTKN2BGX5:1:11101:24202:1051 1:N:0:NTCG
NTCG
+
#EEE
@NB551168:120:HTKN2BGX5:1:11101:4381:1051 1:N:0:NCTT
NCTT
+
#EEA

The output file should contain the following (for each block (4 lines)):

1. The first line exactly as in R2 (it has a space in it, it is important to keep)
2. concatenate the 4 characters (always 4 letters) of the 2nd line from the I file to the beginning of the 2nd line of the R file.
3. keep the 3rd line as the 3rd line in the R file
4. concatenate the 4th lines (any character) in the I file to the beginning of the 4th line in the R file.
5. do it to all blocks

example of an output:

@NB551168:120:HTKN2BGX5:1:11101:3598:1051 2:N:0:NATC
NATCCAATCTCTAAAGTTT
+
#EEEAA/A/EEEE///EEE
@NB551168:120:HTKN2BGX5:1:11101:24202:1051 2:N:0:NTCG
NTCGTGAGACCGGGTGTTG
+
#EEEAAAAAAEEE///<AA
@NB551168:120:HTKN2BGX5:1:11101:4381:1051 2:N:0:NCTT
NCTTGCTACTCCTAAGGCA
+
#EEAA////6/////EE//

Thanks a lot, HM

Answer 1

#!/usr/bin/perl

use strict;

my $f1 = shift;
my $f2 = shift;

open(F1,"<",$f1) || die "couldn't open '$f1' for read: $!\n";
open(F2,"<",$f2) || die "couldn't open '$f2' for read: $!\n";

# set the input record separator (IRS) to '@'
$/='@';

# Normally the IRS is found at the END of a record, but your input
# files START with the input record separator, so we need to throw
# away the first (bogus) input record (i.e. everything from the start
# of the files to the first @ characters in them. in other words, the
# first @ characters in both files.)
my $junk = <F1>;
$junk = <F2>;

while (!eof(F1) && !eof(F2)) {
  my @record1 = split(/\n/, <F1>);
  my @record2 = split(/\n/, <F2>);

  printf "%s%s\n", $/, $record1[0];  # prepend the IRS
  printf "%s%s\n", substr($record2[1],0,4), $record1[1];
  printf "%s\n",   $record2[2];
  printf "%s%s\n", $record2[3], $record1[3];
};

close(F1);
close(F2);

This opens both files for read, and sets perl's $/ input record separator variable to an @ character.

Then, while both files haven't reached EOF, it reads a record from each file, splitting the records into arrays (using newline \n as the delimiter), and then outputs the merged records as specified.

Note that perl arrays start from 0, not 1 - so, e.g., $record1[0] is the first line of the record from file1.

Save the script to a file (e.g. hassan.pl), make it executable with chmod +x hassan.pl and run it as shown below:

Sample output:

$ ./hassan.pl file1.txt file2.txt  
@NB551168:120:HTKN2BGX5:1:11101:3598:1051 2:N:0:NATC
NATCCAATCTCTAAAGTTT
+
#EEEAA/A/EEEE///EEE
@NB551168:120:HTKN2BGX5:1:11101:24202:1051 2:N:0:NTCG
NTCGTGAGACCGGGTGTTG
+
#EEEAAAAAAEEE///<AA
@NB551168:120:HTKN2BGX5:1:11101:4381:1051 2:N:0:NCTT
NCTTGCTACTCCTAAGGCA
+
#EEAA////6/////EE//

(I verified with diff that this output matches your desired output exactly).

Answer 2

I put it in other words: Except for the + line and the line starting with @, paste the I lines before the R lines.

If you see it that why, it gets really simple:

sed '/^[@+]/s/.*//' I|paste -d '' - R

• /^[@+]/ selectes lines starting with + or @
• s/.*// empties those lines
• paste -d '' - R pasted the result (- for the piped input) without delimiter (-d '') to the R file.

Answer 3

If you have access to Gnu sed, you can do this:

$ sed -e 'R I' R | sed -ne 'p;n;n;h;n;G;s/\n//p'