I have 2 input files. Each line of File1 should be compared to each and every line of File2.

Logic is:

  1. If Column1 of File1 does not match Column1 (all the values under it) of File2, then print the whole line of File1 in the output file. Similarly, compare each value of Column1 to each and every value under Column1 of File2.

  2. If Column1 of the two files matches and if the value of Column2 of File1 is greater than N+10 or less than N-10, where N is the value of Column2 of File2, only then print the whole line of File1 and compare all the lines in File2 like this.


Contig1  23
Contig1  42
Contig2  68
Contig3  89
Contig3  102
Contig7  79


Contig1  40
Contig1  49
Contig3  90
Contig2  90
Contig20 200
Contig1  24

Expected Output:

Contig2  68
Contig3  102
Contig7  79

Any solution, even those without awk or sed, will do it.

I found a similar problem, but I'm not exactly sure what I have to do:

Here is the Code:

  `NR==FNR { 
   lines[NR,"col1"] = $1
   lines[NR,"col2"] = $2
   lines[NR,"line"] = $0
   (lines[FNR,"col1"] != $1) {
    print lines[FNR,"line"]
   (lines[FNR,"col2"]+10 < $2 || lines[FNR,"col2"]-10 > $2) {
    print lines[FNR,"line"]
    }' file1 file2`
  • 1
    I don't understand how you want to deal with duplicate entries in file1. For example, you have contig1 with both 23 and 42 in file1 and 40 and 49 both of which are > 23+10. Yet nothing is printed for contig1 in your desired output. – terdon Jul 10 '14 at 14:26
  • Hello Terdon, yes the dataset is correct.So in file1 I am checking Contig1 23 with all the values in File2. Since there is Contig1 24 in file2, which is within the range 23+10 or 23-10, therefore it is not in my output file. For Example, you see Contig2 68 of file1. There is another Contig2 90 in file2. It comes as an output as 90 is not within the range of (68+10) or (68-10). Do I sound clear? – Namrata Jul 10 '14 at 14:47
  • Not really, no. There is no 24 for contig1, did you mean 42? I still don't understand how you want to deal with duplicate values. Your file1 has 2 values for contig1: 23 and 42. Your file2 also has two values: 40 and 49. Which of these do you want to compare? The 23 is not in the range 40-10 - 40 +10 so why is it not printed? Do you only want to compare the greatest values from each file? – terdon Jul 10 '14 at 14:47
  • This might be easier to clarify if you come into our chat. If you do, leave me a message (@terdon) and if I can, I'll try and sort it out with you. – terdon Jul 10 '14 at 14:52
  • Yes there is Contig1 24 in File2. Ok so what I want to do is, I want to compare Contig1 23 of File1 with all the columns of File2 first. Since the last value of File2 is Contig1 24. That is the reason it is not printed since it falls in the range (23+10). Next I compare Contig1 42 of File1 to all the values of File2. Since the first value of File2 is Contig1 40, as 40 belongs to the range (42-10). Next I compare Contig2 68 of File1 to all the values of File2. Since there is Contig2 90, which DOES NOT fall in the range of (68+10) or (68-10). Therefore, I print it. – Namrata Jul 10 '14 at 15:02

The script below does the following, I think this is what you wanted:

  1. If a contig from file1 is not present in file2, print all lines of that contig.
  2. If it is present in file2, then for each value from file1, print it only if it is not less than any of that contig's values from file2 -10 or greater than any of file2's values +10.
#!/usr/bin/env perl

my (%file1, %file2);

## read file1, the 1st argument
    ## Split the line on whitespace into the @F array.
    my @F=split(/\s+/); 

    ## Save all lines in the %file1 hash.
    ## $F[0] is the contig name and $F[1] the value.
    ## The hash will store a list of all values
    ## associated with this contig.
    push @{$file1{$F[0]}},$F[1];
## read file2, the second argument
    ## remove newlines
    ## save the fields into array @F
    my @F=split(/\s+/); 
    ## Again, save all values associated with each
    ## contig into the %file2 hash. 
    push @{$file2{$F[0]}},$F[1];

## For each of the contigs in file1
foreach my $contig (keys(%file1)) {
    ## If this contig exists in file 2
    if(defined $file2{$contig}){
        ## get the list of values for that contig
        ## in each of the two files
        my @f2_vals=@{$file2{$contig}};
        my @f1_vals=@{$file1{$contig}};
        ## For each of file1's values for this contig
        val1:foreach my $val1 (@f1_vals) {
                ## For each of file2's value for this contig
                foreach my $val2 (@f2_vals) {
                    ## Skip to the next value from file1 unless
                    ## this one falls within the desired range.
                    unless(($val1 < $val2-10) || ($val1 > $val2+10)){
                        next val1;
                ## We will only get here if none of the values
                ## fell within the desired range. If so, we should
                ## print the value from file1.
                print "$contig $val1\n";
    ## If this contig is not in file2, print the
    ## lines from file1. This will print all lines
    ## from file1 whose contig was not in file2.
    else {
        print "$contig $_\n" for @{$file1{$contig}}

Save this in a text file (say foo.pl), make it executable (chmod a+x foo.pl) and run it like this:

./foo.pl file1 file2

On your example, it returns:

$ foo.pl file1 file2 
Contig2 68
Contig3 102
Contig7 79
  • Thanks very much it works and giving me the correct output. – Namrata Jul 11 '14 at 13:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.