how to delete consecutive records if they have the same field using awk or sed?

Question

Hi I want to delete the lines which contains the same numbers, how to achieve that in awk or sed or perl? For example, line 4 and 5 both contains 12.7, so I want both lines to be removed. Thanks.

sample input

start=0.1
end=2.5
start=8.7
end=12.7
start=12.7
end=16.7
start=16.7
end=25.2
start=25.2
end=48.7
start=48.7
end=60.1
start=66.2
end=69.2
start=69.2
end=72.2
start=72.2
end=75.2
start=75.2
end=78.2

expected output:

start=0.1
end=2.5
start=8.7
end=60.1
start=66.2
end=78.2

Answer 1

With the GNU implementation of uniq, and assuming the input doesn't contain TAB characters, you could do:

<your-file tr = '\t' | uniq -uf1 | tr '\t' =

Answer 2

With awk double parsing the file and printing only lines with unique last field.

$ awk -F= 'p==1{seen[$NF]++} p==2 && seen[$NF]==1' p=1 file p=2 file
start=0.1
end=2.5
start=8.7
end=60.1
start=66.2
end=78.2

Or the same, using the classic awk idiomatic syntax (FNR==NR stands for 'when reading the first file', also next is necessary)

awk -F= 'FNR==NR {seen[$NF]++; next} seen[$NF]==1' file file

Answer 3

Ignoring the line numbers (which I believe are just there for the sake of the readers of the question), you seem to basically want to merge ranges that lie back to back, i.e. remove any end=X start=Y pair where X and Y are the same number.

$ tr '\n' '\t' < file | sed 's/end=\([[:digit:].]*\)[[:blank:]]start=\1[[:blank:]]//g' | tr '\t' '\n'
start=0.1
end=2.5
start=8.7
end=60.1
start=66.2
end=78.2

This first replaces each newline in the input data with a tab, and then uses sed to remove each end= start= pair that have the same number after the = sign. Once the ranges have been merged, the tabs are replaced by newlines again.

Answer 4

Also ignoring the line numbers, here is an awk-based solution:

awk '{split($1,a,"=");
      if (a[1]=="start") {st=a[2]; if (st!=en) {if (en) print last; print; last=""}};
      if (a[1]=="end") {en=a[2]; last=$0}}
     END{if (last) print last}' input.txt

• This will parse the lines to record start end end times, but print nothing by default.
• If an "end" statement is found, the line is stored in last.
• If a "start" line is found, and the start time is different from the last end time, both the last "end" line and the current "start" line are printed, and the "last" buffer is cleared.
• If at end of file a "last" line remains (which should be the usual case), it is also printed.

Answer 5

for i in `awk -F "=" '{if(!seen[$NF]++)print $NF}' file`; do   p=`awk -F "=" -v i="$i" '$NF == i{print $0}' file|wc -l`;if [[ $p == 1 ]]; then awk -F "=" -v i="$i" '$NF == i{print $0}' file; fi; done

output

start=0.1
end=2.5
start=8.7
end=60.1
start=66.2
end=78.2

Answer 6

GNU sed: maintain a 2-line pattern space. Completely delete pattern space if a match is found otherwise print the whole.

sed -Ee '
  /^end=/!b
  $!N
  /=(.*)\nstart=\1$/d
' file 

perl -aF= -pe '
  next if eof || !/^end=/;
  ($_ .= <>) =~ /=(\S+)$/;
  s/.*//s if  $1 == $F[1];
' file 

awk -F '=' '
  BEGIN { OFS = ORS }
  END { if (f) print l }
  /^end=/ && !f {
    l = $0; t = $2
    f = 1; next
  }
  /^start=/ && f {
    if ($2+0 != t) print l, $0
    f = 0
    next
  }1
' file

Output:

start=0.1
end=2.5
start=8.7
end=60.1
start=66.2
end=78.2