0

FileA.txt:

ATGCATGC 
GGGGGGTT
TTTTT
AAAA

FileB.txt:

asdfasdf
blah2
ATGCATGC
blah3
blah4 
delte-me-too
GGGGGGTT
blah5
blah5
....

I want to compare the each line from FileA.txt and check if it is in FileB.txt. If it is in FileB, I want to delete the following:

  1. Matched Line

  2. One line above

  3. Two lines below

and output into a new file.

NOTE: There will be 500,000 lines in FileA. I would like to do this in a way which we do not hardcode the patterns.

I currently have something to delete the lines, but I'm getting tripped up about the looping through FileA to create a new pattern for this awk expression:

awk '/$VARIABLE_REGEX/{for(x=NR-2;x<=NR+2;x++)d[x];} {a[NR]=$0}
END{for(i=1;i<=NR;i++)
    if(!(i in d))
        print a[i]}' FileB.txt
0

Note: No error checking. Also, assumption is that the input in 2nd file follows the pattern mentioned exactly.

awk 'NR== FNR {a[$0] = $0 ; next } {if (!($0 in a)) {b[count++] = $0; } else {count--; if (count > 0) delete b[count];getline;getline; }} END {for (i=0; i<count; i++) print b[i] }' 1 2

inputs are in 1 and 2

1

ATGCATGC
GGGGGGTT
TTTTT
AAAA

2

asdfasdf
blah2
ATGCATGC
blah3
blah4 
delte-me-too
GGGGGGTT
blah5
blah5
foo
foo-delete
AAAA
bar-delete
bar-delete
bar-ok

Output

asdfasdf
foo
bar-ok
| improve this answer | |
  • Thanks for the amazing response! The awk command works perfectly and the method is very clear. Much appreciated! – Anonymous Feb 6 '16 at 7:40
0

Next code is not optimal (because it have to read FileB.txt twice) but hope it can be more quick then awk

comm --nocheck-order -23 FileB.txt <(grep -B1 -A2 -Ff FileA.txt FileB.txt)

With new GNU sed command e you can try (for memory saving) sed + grep:

sed 'N;h;s/.*\n//;s/.*/grep -xF "&" FileA.txt/e;/./{N;N;d};x;P;D' FileB.txt
| improve this answer | |
0

This works with your sample

awk '
    NR==FNR {patt[$0]; next} 
    $0 in patt {getline; getline; getline; prev=$0; next} 
    {print prev; prev=$0} 
    END {print prev}
' fileA.txt fileB.txt 

You have to keep all of fileA in memory, but you only need to remember one line at a time from fileB

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.