0

How to compare consecutive lines from a text file where last column should be same? And at least one value in the 2nd last column must be mismatched.

  1. Firstly, column 6 should be compared in the file, if matched
  2. Then Compare column 4 for matched lines. In the column 4 all values should not be same at least one should be different. Here HSF1 and HIF1AN in line number 1,2,3.

INPUT FILE

chr10   102979  103832  HSF1        305
chr10   102979  103832  HIF1AN  1   305
chr10   102979  103832  HSF1        305
chr10   103124438   103124851   HSF1        471
chr10   103124438   103124851   EGLN1   2   471
chr10   103969896   103970503   HSF1        472
chr10   103969896   103970503   HSF1        472
chr10   39135037    39142175    HSF1        335
chr10   4191461 4191936 HSF1        309
chr10   4191461 4191936 HSF1        309
chr10   42423355    42424014    HSF1        336

OUTPUT FILE

chr10   102979  103832  HSF1        305
chr10   102979  103832  HIF1AN  1   305
chr10   102979  103832  HSF1        305
chr10   103124438   103124851   HSF1        471
chr10   103124438   103124851   EGLN1   2   471
  • 2
    What is your approach on this, e.g. what have you tried so far? – maulinglawns Sep 6 '18 at 15:39
  • What is the field separator here? It appears that some lines have 5 fields, some have 6. Unless it is tab-separated. – glenn jackman Sep 6 '18 at 16:03
  • I see you are actually removing those lines having same column4&6; else print all those lines if another line found with same column6 but different in column4. – αғsнιη Sep 6 '18 at 18:37
  • Maulinglawns: Thanks for your participation. I am trying but not able to make the logic with awk. I am a biologist. However, need it for my gradutation project. – Himanshu Singh Sep 7 '18 at 9:52
  • Glenn Jackman: It is tab separated file. Yes, in few lines column 5 is empty. – Himanshu Singh Sep 7 '18 at 9:53
2

Assuming a tab-separated file, you can use GNU awk:

gawk -F'\t' 'NR == FNR {count[$6][$4]++; next} length(count[$6]) > 1' file file
chr10   102979  103832  HSF1        305
chr10   102979  103832  HIF1AN  1   305
chr10   102979  103832  HSF1        305
chr10   103124438   103124851   HSF1        471
chr10   103124438   103124851   EGLN1   2   471

This passes through the file twice: the first time to count how many $4 values occur for each $6; the second time to output a record if there are more than 1 $4 values for that $6.

It could probably be done in a single pass, at the expense of complexity, memory usage and possibly the loss of the original ordering.


Same logic in perl

perl -Mautodie -e '
    open $f, "<", shift;
    while (<$f>) {
        @F = split /\t/;
        $c{ $F[5] }{ $F[3] }++;
    }
    # re-process the file
    seek $f, 0, 0;
    while (<$f>) {
        @F = split /\t/;
        print if scalar keys %{ $c{$F[5]} } > 1;
    }
' file
| improve this answer | |
  • gawk: NR == FNR {count[$6][$4]++; next} length(count[$6]) > 1 gawk: ^ syntax error – Stack EG Sep 6 '18 at 17:17
  • Am getting syntax error before [$4] – Stack EG Sep 6 '18 at 17:17
  • what version do you have? gawk --version – glenn jackman Sep 6 '18 at 17:40
  • GAWK version is GNU Awk 3.1.7 – Stack EG Sep 6 '18 at 17:41
  • 2
    suggestion: you can use $NF in count[$NF][$4]++ instead and length(count[$NF]) , then you can remove the -F'\t' if we don't know what is the separator. – αғsнιη Sep 6 '18 at 18:38
0

Read the file line by line, store the line in a buffer if the last column has the same value as all the other lines in the buffer, otherwise process the buffer and empty it.

Processing the buffer means trying to find at least one pair of consecutive lines that don't share the 4th column's value and printing the buffer if successful.

I stored the exact lines in the buffer together with the individual columns for nicer output and easier access to the values.

#!/usr/bin/perl
use warnings;
use strict;

sub process {
    my (@rows) = @_;
    my $different;
    for my $i (1 .. $#rows) {
        $different = 1, last if $rows[ $i - 1 ][4] ne $rows[$i][4];
    }
    print map "$_->[0]\n", @rows if $different;
}

my @buffer;
while (<>) {
    chomp;
    my @columns = split;
    if (! @buffer || $buffer[0][-1] == $columns[-1]) {
        push @buffer, [$_, @columns];
    } else {
        process(@buffer);
        @buffer = [$_, split];
    }
}
process(@buffer);
| improve this answer | |
  • Thank you for sharing the code. Being biologist not able to understand much. Please explain !@buffer in the code? Please add one more criteria in the code. I would highly appreciate. Criteria: 1. Firstly, column 6 should be compared in the file, if matched Modified Criteria: 2. Then Compare column 4 for matched lines. In the column 4 all values should not be same at least one HSF1 must be there. Here HSF1 and HIF1AN in line number 1,2,3. – Himanshu Singh Sep 7 '18 at 11:54
  • ! is the negation operator, so ! @buffer is true when the array @buffer is empty. – choroba Sep 7 '18 at 15:19
  • To compare other columns, just use the same trick as I used to check there are different values in column 4, $rows[ $i - 1 ] == $rows[$i] must be true for all $i. Checking that column 4 has at least two different values is already implemented, so just add a test that it contains the needed value somewhere. – choroba Sep 7 '18 at 15:40
  • 1
    Thank you very much for supporting me. I really appreciate your help. The script is working as the table I want. Thanks again. – Himanshu Singh Sep 10 '18 at 12:30
  • 1
    "Thank you" on StackExchange = upvote/accept. – choroba Sep 10 '18 at 17:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.