Keep only the first line from every sequence of consecutive lines matching a pattern

Question

If 2 or more consecutive lines contain a specific pattern then delete all matching lines and keep just the first line.

In below example when 2 or more consecutive lines contain "logical IO" then we need to delete all matching lines but keep the first line.

Input file:

select * from test1 where 1=1
testing logical IO 24
select * from test2 where condition=4
parsing logical IO 45
testing logical IO 500
handling logical IO 49
select * from test5 where 1=1
testing logical IO 24
select * from test5 where condition=78
parsing logical IO 346
testing logical IO 12

Output file:

select * from test1 where 1=1
testing logical IO 24
select * from test2 where condition=4
parsing logical IO 45
select * from test5 where 1=1
testing logical IO 24
select * from test5 where condition=78
parsing logical IO 346

Answer 1

Using awk:

awk '/logical IO/ {if (!seen) {print; seen=1}; next}; {print; seen=0}' file.txt 

• /logical IO/ {if (!seen) {print; seen=1}; next} checks if the line contains logical IO, if found and the variable seen is false i.e. previous line does not contain logical IO, then print the line, set seen=1 and go to the next line else go to the next line as the previous line has logical IO

• For any other line, {print; seen=0}, prints the line and the sets seen=0

Example:

$ cat file.txt 
select * from test1 where 1=1
testing logical IO 24
select * from test2 where condition=4
parsing logical IO 45
testing logical IO 500
select * from test5 where 1=1
testing logical IO 24
select * from test5 where condition=78
parsing logical IO 346
parsing logical IO 346
testing logical IO 12

$ awk '/logical IO/ {if (!seen) {print; seen=1}; next}; {print; seen=0}' file.txt 
select * from test1 where 1=1
testing logical IO 24
select * from test2 where condition=4
parsing logical IO 45
select * from test5 where 1=1
testing logical IO 24
select * from test5 where condition=78
parsing logical IO 346

Answer 2

With sed:

sed '/logical IO/{x;//!{g;p;};d;};//!h' infile

how it works:

sed '/logical IO/{         # if line matches
x                          # exchange hold space w. pattern space
//!{                       # if whatever was in the hold buffer doesn't match
g                          # overwrite pattern space with hold space content
p                          # print current pattern space
}
d                          # delete
}
//!h                       # if line doesn't match, copy over the hold space
' infile

Answer 3

In the TXR language we can express this without any mutating state variables. At any given position in the file we can perform a multi-line pattern match with two branch alternatives: we either match one or more consecutive lines which contain the search string and then print the first, or else we match one line and print that. One possible way is this:

@(repeat)
@  (cases)
@    (collect :gap 0 :mintimes 1)
@line
@      (require (search-str line "logical IO"))
@    (end)
@    (do (put-line (first line)))
@  (or)
@line
@    (do (put-line line))
@  (end)
@(end)

Run:

$ txr first-log-IO.txr data
select * from test1 where 1=1
testing logical IO 24
select * from test2 where condition=4
parsing logical IO 45
select * from test5 where 1=1
testing logical IO 24
select * from test5 where condition=78
parsing logical IO 346

@(repeat) establishes a walk through the data without collecting variable bindings; when this construct is seen it usually indicates that some side-effect takes place in the iteration. In this case it is output.

Inside the @(repeat) we have a @(cases) construct: multi-way match consisting of cases separated by @(or). The second branch, the fallback case, of this is just @line which matches a line. The @(do (put-line (first line))) directive which follows prints that line.

The main branch of the @(cases) collects material via @(collect). The matches must be consecutive, required by :gap 0, and there must be at least one, required by :mintimes 1. The collect body matches a single line, bound to the line variable. Then there is a @(require ...) assertion which fails unless the line contains the substring "logical IO". The collection will therefore stop when it encounters a non-matching line, because the :gap 0 prevents it from skipping it. The matching lines are implicitly collected into a list called line which pops out of the collect (a variable bound inside a collect automatically becomes a list outside of the collect, of all the values bound over the multiple iterations). We just print the first one, as required, suppressing the rest.

Note that the two @line matches have nothing to do with each other; they bind the line variable in different scopes.

Another way is to do some functional programming over lazy lists in TXR Lisp:

[(opip (partition-by (do cond
                       ((search-str @1 "logical IO") t)
                       (t @1)))
       (mapcar* first)
       put-lines)
 (get-lines)]

$ txr first-log-IO.tl

The opip operator is a syntactic sugar for constructing a pipeline of functions. Its arguments are all treated as op syntax: a macro for generating anonymous functions with implicit, numbered arguments.

The overall form is [(opip ...) (get-lines)] which just means "call the function produced by opip, with the result of (get-lines) as an argument". This (get-lines) converts the standard input stream into a lazy list of strings. (Its "opposite" is put-lines, which makes an appearance).

Now in the pipeline, we use partition-by to (lazily!) convert the list of lines into a list of lists which are its partitions. The partitioning condition is such that each line which contains logical IO is mapped to the symbol t, and all other lines just map to themselves. This means that consecutive lines which contain logical IO appear as a partition, and all other lines appear isolated as partitions of length one. All we have to do with this data now is map each partition to its first item, via (mapcar* first) and pass it to put-lines to dump the result.

We use mapcar* because that's the lazy version of mapcar. We want everything to be lazy so that the action is actually triggered by put-lines. As put-lines marches through the output list, it pulls items out of the lazy mapcar*, which forces the elements produced by the partition-by, which forces the list produced by (get-lines) which causes I/O to take place to read those lines.

If we use the regular mapcar by mistake, we cause the problem that the entire output is constructed in memory before being dumped out, which bodes badly for large files.