Comparing two files linewise and if pattern of file 1 is not found (fully or partially) in file 2 then print line of file 1

Question

I have two files:

file1 (search):

1  
GACGGAGGATGCAAGTGTTATCCGGAATCACTGGGCGTAAAGTTTTTTTTT  
2  
GACGGAGGATGCAAGTGTTATCCGGAAT  
3  
GACGGAGGATGCAAGTGTTATCCGGAATCACTGGGCGTAAAGCGTCC  
4  
GACGGAGGATGCAAGTGTTATCCGGAATCACTGGGCCGTCCGTAG 

file2 (patterns):

GACGGAGGATGCAAGTGTTATCCGGAATCACTGGGCGTAAAGC
GACGGAGGATGCAAGTGTTATCCGGAATCACTGGGCGTAAAGCG
GACGGAGGATGCAAGTGTTATCCGGAATCACTGGGCGTAAAGCGTCC
GACGGAGGATGCAAGTGTTATCCGGAATCACTGGGCGTAAAGCGTCCGTAG 

What I need is:

1  
GACGGAGGATGCAAGTGTTATCCGGAATCACTGGGCGTAAAGTTTTTTTTT  
2  
3  
4  
GACGGAGGATGCAAGTGTTATCCGGAATCACTGGGCCGTCCGTAG  

Which means, that I need a script that searches my search-lines in the patterns-file, and if the search-lines can be found fully or partially (as for the line following "2") then they should not be written, but all other lines of file 1.

I tried many of the grep and awk scripts found in that forum but none did what I need, e.g.

awk 'FN==NR {exclude[$0];next} !($0 in exclude)' file2 file1  

or

awk 'NR==FNR{a[$1]++;next} !($1 in a) {print $1} {next}' file2 file1

I also tried grep.

Anyway, all tried scripts just find the fully matching patterns but not the partially matching ones...

Has anyone have an idea??

Answer 1

Using an awk script:

NR == FNR       { seq[++n] = $1; next }

{
    header = $0
    getline

    for (i = 1; i <= n; ++i) {
        if (match(seq[i], $0) > 0) {
            print header
            next
        }
    }

    print header
    print
}

Running it:

$ awk -f script.awk file2 file1
1
GACGGAGGATGCAAGTGTTATCCGGAATCACTGGGCGTAAAGTTTTTTTTT
2
3
4
GACGGAGGATGCAAGTGTTATCCGGAATCACTGGGCCGTCCGTAG

The script first reads the 2000 sequences from file2 into the array seq, and then reads the header from file1 followed by the sequence from file1 (using getline). It then goes through the array seq to find a sequence that contains the current sequence from file1. If such a sequence is found, the header is printed and the script continues with the next line in file1. Otherwise, both header and sequence are printed.

The script assumes that the file1 contains alternating header and sequence lines, and absolutely no multi-line sequence.

The following is the same script but using DRY principle ("Don't repeat yourself"):

NR == FNR       { seq[++n] = $1; next }

{
    header = $0
    getline

    found = 0
    for (i = 1; i <= n; ++i) {
        if (match(seq[i], $0) > 0) {
            found = 1
            break
        }
    }

    print header
    if (!found) print
}

Answer 2

awk + grep solution:

awk '/^[^>0-9]/{ 
         m = ""; cmd = sprintf("grep -m1 %s file2", $0);
         cmd | getline m; close(cmd);
         if (m) next; 
     }1' file1

• cmd - crucial grep command
• m - variable filled with possible matched item

The output:

1
GACGGAGGATGCAAGTGTTATCCGGAATCACTGGGCGTAAAGTTTTTTTTT
2
3
4
GACGGAGGATGCAAGTGTTATCCGGAATCACTGGGCCGTCCGTAG

Answer 3

grep -vxf <(grep -of file_1 file_2 | sort -u) file_1

file_1

1
GACGGAGGATGCAAGTGTTATCCGGAATCACTGGGCGTAAAGTTTTTTTTT
2
GACGGAGGATGCAAGTGTTATCCGGAAT
3
GACGGAGGATGCAAGTGTTATCCGGAATCACTGGGCGTAAAGCGTCC
4
GACGGAGGATGCAAGTGTTATCCGGAATCACTGGGCCGTCCGTAG

file_2

GACGGAGGATGCAAGTGTTATCCGGAATCACTGGGCGTAAAGC
GACGGAGGATGCAAGTGTTATCCGGAATCACTGGGCGTAAAGCG
GACGGAGGATGCAAGTGTTATCCGGAATCACTGGGCGTAAAGCGTCC
GACGGAGGATGCAAGTGTTATCCGGAATCACTGGGCGTAAAGCGTCCGTAG

Output

1
GACGGAGGATGCAAGTGTTATCCGGAATCACTGGGCGTAAAGTTTTTTTTT
2
3
4
GACGGAGGATGCAAGTGTTATCCGGAATCACTGGGCCGTCCGTAG