3

I have log files with a time stamp and six values in each line i want to reduce the amount of data, by removing consecutive lines with the same values (ignoring time stamps) and keeping the first and last line of each duplicate set. Preferably using a bash script. It should be a magic sed or awk command combination.

Even if i have to parse the file multiple times, reading 3 lines at a time and removing the middle one, is a good solution.

original file:

1447790360      99999   99999   20.25   20.25   20.25   20.50
1447790362      20.25   20.25   20.25   20.25   20.25   20.50
1447790365      20.25   20.25   20.25   20.25   20.25   20.50
1447790368      20.25   20.25   20.25   20.25   20.25   20.50
1447790371      20.25   20.25   20.25   20.25   20.25   20.50
1447790374      20.25   20.25   20.25   20.25   20.25   20.50
1447790377      20.25   20.25   20.25   20.25   20.25   20.50
1447790380      20.25   20.25   20.25   20.25   20.25   20.50
1447790383      20.25   20.25   20.25   20.25   20.25   20.50
1447790386      20.25   20.25   20.25   20.25   20.25   20.50
1447790388      20.25   20.25   99999   99999   99999   99999
1447790389      99999   99999   20.25   20.25   20.25   20.50
1447790391      20.00   20.25   20.25   20.25   20.25   20.50
1447790394      20.25   20.25   20.25   20.25   20.25   20.50
1447790397      20.25   20.25   20.25   20.25   20.25   20.50
1447790400      20.25   20.25   20.25   20.25   20.25   20.50

desired result:

1447790360      99999   99999   20.25   20.25   20.25   20.50
1447790362      20.25   20.25   20.25   20.25   20.25   20.50
1447790386      20.25   20.25   20.25   20.25   20.25   20.50
1447790388      20.25   20.25   99999   99999   99999   99999
1447790389      99999   99999   20.25   20.25   20.25   20.50
1447790391      20.00   20.25   20.25   20.25   20.25   20.50
1447790394      20.25   20.25   20.25   20.25   20.25   20.50
1447790400      20.25   20.25   20.25   20.25   20.25   20.50
1

With awk one liner:

awk '{n=$2$3$4$5$6$7}l1!=n{if(p)print l0; print; p=0}l1==n{p=1}{l0=$0; l1=n}END{print}' file

The whole point is to manipulate few variables: n stores all fields except first in current line, l1 the same for previous line and l0 the whole previous line. The p is just a flag to mark if previous line was already printed.

0

Perl to the rescue:

perl -ne '($t, $r) = /([0-9]+\s+)(.*)/;
          print "$pt$p\n$_" if $r ne $p;
          $p = $r;
          $pt = $t;
          }{
          print $t, $r' input-file \
| sort -nu | tail -n+2
  • -n reads the input line by line.
  • $t is the timestamp plus whitespace, $r is the "rest".
  • $p is the previous rest, $pt is the previous timestamp.
  • the last line is always printed

Perl prints some of the lines twice, sort -nu should remove the duplicates. tail removes the first empty line.

0

A awk version:

#!/usr/bin/gawk -f

BEGIN                      { rep=0  ; prev="" ; OFS=FS="     "}
!rep && prev== $2          { rep=1  ; prev=$2 ;          next }
rep  && prev== $2          { time=$1 ;                   next }
rep  && prev!= $2          { rep=0  ; print time, prev;       }
                           { print  ; prev=$2                 }
END                        { if(rep){ print time, prev}       }
  • rep = inside a repetition?
  • prev = previous number values
  • time = previous time-stamp
0
sed -e:t -e'$!{1N;N;s/\( .*\)\(\n[^ ]*\1\)\{2\}$/\1\2/;tt' -e'P;D;}' <in >out

...that works. it recursively substitutes away the second in a series of three of input lines which it deems to be identical from the second space-delimited field on. it will continue to draw another input line to replace each it substitutes away until it can no longer do so. only when it cannot match three such similar lines will it Print the first in its buffer before Deleting it and cycling back to try again with the two that remain and the next line of input.

with GNU or BSD sed:

sed -Ee:t -e'1N;$!N;s/( .*)(\n[^ ]*\1){2}$/\1\2/;tt' -eP\;D <in >out

1447790360      99999   99999   20.25   20.25   20.25   20.50
1447790362      20.25   20.25   20.25   20.25   20.25   20.50
1447790386      20.25   20.25   20.25   20.25   20.25   20.50
1447790388      20.25   20.25   99999   99999   99999   99999
1447790389      99999   99999   20.25   20.25   20.25   20.50
1447790391      20.00   20.25   20.25   20.25   20.25   20.50
1447790394      20.25   20.25   20.25   20.25   20.25   20.50
1447790400      20.25   20.25   20.25   20.25   20.25   20.50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.