3
2018-05-24 23:57:30 1.1.1.1 8.8.4.4
2018-05-24 23:57:32 2.2.2.2 8.8.4.4
2018-05-24 23:58:12 8.8.8.8 8.8.4.4
2018-05-24 23:58:23 8.8.8.8 8.8.4.4
2018-05-24 23:59:40 8.8.8.8 8.8.4.4
2018-05-24 23:59:51 8.8.8.8 8.8.4.4

As I've got the log file in above format. Now I need to parse it and output should look like below(Showing only First and the Last line by comparing the 3rd and 4th column if the row data is repeated.

2018-05-24 23:57:30 1.1.1.1 8.8.4.4
2018-05-24 23:57:32 2.2.2.2 8.8.4.4
2018-05-24 23:58:12 8.8.8.8 8.8.4.4
2018-05-24 23:59:51 8.8.8.8 8.8.4.4
1
  • 1
    Do you have any other lines where the 3rd and 4th column are not the same or do you just want to print the first and last line? – jesse_b May 25 '18 at 16:13
4

With awk:

awk '!first[$3, $4]{ first[$3, $4]= $0 } { last[$3, $4]= $0 }
    END{ for (x in last) print first[x] (last[x] != first[x]? ORS last[x]:"") }' infile
2018-05-24 23:58:12 8.8.8.8 8.8.4.4
2018-05-24 23:59:51 8.8.8.8 8.8.4.4
2018-05-24 23:57:30 1.1.1.1 8.8.4.4
2018-05-24 23:57:32 2.2.2.2 8.8.4.4

the first associated array keeps the first occurred line with the keys combination of column#3 and column#4, but the last array keeps holding the latest line with those same keys each time.

After all lines were read, the values in first array are the lines which occurred at very first (with different column#3, #4) and the values in last are the lines which occurred at very last.

Then at the END print the values saved in first array and next in last. This (last[x] != first[x]? ORS last[x]:"") is used to prevent duplicaing the line when that's the only unique line with no repeated column3&4 combination.

6

Perl to the rescue:

perl -ane '
    if ($F[2] ne $c3 || $F[3] ne $c4) {
        $printed or print $previous;
        $printed = print;
    } else {
        $printed = 0;
    }
    ($c3, $c4, $previous) = (@F[2, 3], $_);
    END { print $previous unless $printed }
' -- input.file
  • -n reads the input line by line and runs the code for each line.
  • -a splits each input line on whitespace into the @F array.
  • $c3 and $c4 are used to keep the previous values of columns 3 and 4, the actual values are stored in $F[2] and $F[3] (arrays are indexed from 0).
  • $previous stores the previous line in case we needed to print it.
  • $printed just prevents printing the last line twice (which would otherwise happen if its columns 3 and 4 are different to the previous line).
0
0

You could also just take unique lines only comparing columns 3,4 and then append the last line in this case. But this might result in a last line duplicate if all other lines have different 3rd and 4th columns.

Then just add another pipe into uniq to remove if needed.

{uniq <your_file> -f2; tail -n1 <your_file>; } | cat | uniq

-f here skips the first 2 space delimited fields.

1
  • I think, OP want solution, which process multiple blocks of rows with the same 3,4 columns, not the single. Look at the example of Input in my answer. By the way, the cat in your command doesn't needed. – MiniMax May 26 '18 at 13:05
0
 perl -lane '
   *x = sub { print for splice @A; } if $. == 1;
   x() if $. > 1 and $F[2] ne $c3 || $F[3] ne $c4;
   ($c3, $c4, $A[!!@A]) = (@F[2,3], $_);
   x() if eof;
 '    include.txt

§ How it works.

    °  Array @A holds only 2 elements max at any time. The beginning and end lines for the range.

   °  subroutine &x displays the array @A and after displaying empties it as well.

  °  display the previous range provided we are not at the first line and either of the previous columns don't match with the current.

  °   update the previous columns and array. 
0

First variant

paste -d'\n' <(uniq -f2 input.txt) <(tac input.txt | uniq -f2 | tac) | uniq

Second variant

awk '
$3$4 == prev {
    buf = $0 ORS
}
$3$4 != prev {
    print buf $0
    prev = $3$4
    buf = ""
}
END {
    printf("%s", buf)
}' input.txt

Testing

Input (complicated for testing)

2018-05-24 23:57:30 1.1.1.1 8.8.4.4
2018-05-24 23:57:32 2.2.2.2 8.8.4.4
2018-05-24 23:58:12 8.8.8.8 8.8.4.4
2018-05-24 23:58:23 8.8.8.8 8.8.4.4
2018-05-24 23:59:40 8.8.8.8 8.8.4.4
2018-05-24 23:59:51 8.8.8.8 8.8.4.4
2018-05-25 00:18:12 8.8.1.8 8.8.4.4
2018-05-25 00:18:23 8.8.1.8 8.8.4.4
2018-05-25 00:19:40 8.8.1.8 8.8.4.4
2018-05-25 00:19:51 8.8.1.8 8.8.4.4
2018-05-25 00:39:51 8.8.2.8 8.8.4.4
2018-05-25 00:49:52 8.8.2.8 8.8.4.4
2018-05-25 00:59:51 8.8.2.8 8.8.4.4

Output (both variants)

2018-05-24 23:57:30 1.1.1.1 8.8.4.4
2018-05-24 23:57:32 2.2.2.2 8.8.4.4
2018-05-24 23:58:12 8.8.8.8 8.8.4.4
2018-05-24 23:59:51 8.8.8.8 8.8.4.4
2018-05-25 00:18:12 8.8.1.8 8.8.4.4
2018-05-25 00:19:51 8.8.1.8 8.8.4.4
2018-05-25 00:39:51 8.8.2.8 8.8.4.4
2018-05-25 00:59:51 8.8.2.8 8.8.4.4
0
 perl -lane '
   *x = sub { print for splice @A; } if $. == 1;
   x() if $. > 1 and $F[2] ne $c3 || $F[3] ne $c4;
   ($c3, $c4, $A[!!@A]) = (@F[2,3], $_);
   x() if eof;
 '    include.txt

§ How it works.

    °  Array @A holds only 2 elements max at any time. The beginning and end lines for the range.

   °  subroutine &x displays the array @A and after displaying empties it as well.

  °  display the previous range provided we are not at the first line and either of the previous columns don't match with the current.

  °   update the previous columns and array.

¶ Another method is elaborated which uses the sed editor.

 #! /bin/sh
  # declare regex assist variables
   b='[:space:]'
   s="[$b]"         # \s
   S="[^$b]"       # \S

   #      \S+                \s+
   F="$S$S*"   sp="$s$s*"
   F_s="$F$sp"      #  \S+\s+

   # composition of a line 
   L="$F_s$F_s\($F\)$sp\($F\)"

   #  matching next line
   M=".*$s\1$sp\2"

   #    2 lines when they match with 3,4 fields
   L2="$L\(\\n$M\)\{1\}"

   # 3 lines when they match in fields 3,4
   L3="$L\(\\n$M\)\{2\}"

  #### code 
  sed -e '
       #  bring on board next line for interrogation 
        N

         #   2 lines fields 3,4 donot match 
          #  display the first line... redo code with remaining 
         '"/^$L2\$/"'!{
                  P;D
           }

            #  3 lines with first two match but third not match in fields 3,4
           :a;h;N
           '"/^$L3\$/"'!{
                 x;p;g
                 s/.*\(\n\)/\1/;D
              }

              s/\n.*\(\n\)/\1/;ba
      '   include.txt 

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.