Keep only the first line from every sequence of consecutive lines matching a pattern

If 2 or more consecutive lines contain a specific pattern then delete all matching lines and keep just the first line.

In below example when 2 or more consecutive lines contain "logical IO" then we need to delete all matching lines but keep the first line.

Input file:

``````select * from test1 where 1=1
testing logical IO 24
select * from test2 where condition=4
parsing logical IO 45
testing logical IO 500
handling logical IO 49
select * from test5 where 1=1
testing logical IO 24
select * from test5 where condition=78
parsing logical IO 346
testing logical IO 12
``````

Output file:

``````select * from test1 where 1=1
testing logical IO 24
select * from test2 where condition=4
parsing logical IO 45
select * from test5 where 1=1
testing logical IO 24
select * from test5 where condition=78
parsing logical IO 346
``````
• What if you have 3 or more consecutive lines matching the pattern ? – don_crissti Sep 5 '16 at 10:32
• Nice question. In case we have 3 or more consecutive lines matching the pattern then just keep the first line and delete the others :) Question will be adapted now. – ekassis Sep 5 '16 at 10:37

Using `awk`:

``````awk '/logical IO/ {if (!seen) {print; seen=1}; next}; {print; seen=0}' file.txt
``````
• `/logical IO/ {if (!seen) {print; seen=1}; next}` checks if the line contains `logical IO`, if found and the variable `seen` is false i.e. previous line does not contain `logical IO`, then print the line, set `seen=1` and go to the next line else go to the next line as the previous line has `logical IO`

• For any other line, `{print; seen=0}`, prints the line and the sets `seen=0`

Example:

``````\$ cat file.txt
select * from test1 where 1=1
testing logical IO 24
select * from test2 where condition=4
parsing logical IO 45
testing logical IO 500
select * from test5 where 1=1
testing logical IO 24
select * from test5 where condition=78
parsing logical IO 346
parsing logical IO 346
testing logical IO 12

\$ awk '/logical IO/ {if (!seen) {print; seen=1}; next}; {print; seen=0}' file.txt
select * from test1 where 1=1
testing logical IO 24
select * from test2 where condition=4
parsing logical IO 45
select * from test5 where 1=1
testing logical IO 24
select * from test5 where condition=78
parsing logical IO 346
``````
• Nice. I like that you avoided code golf and used a human readable variable name, `seen`. – Wildcard Sep 5 '16 at 10:52
• @heemayl Thank you! It is working perfectly but with nawk – ekassis Sep 5 '16 at 11:07

With `sed`:

``````sed '/logical IO/{x;//!{g;p;};d;};//!h' infile
``````

how it works:

``````sed '/logical IO/{         # if line matches
x                          # exchange hold space w. pattern space
//!{                       # if whatever was in the hold buffer doesn't match
g                          # overwrite pattern space with hold space content
p                          # print current pattern space
}
d                          # delete
}
//!h                       # if line doesn't match, copy over the hold space
' infile
``````
• I got below error. Too many {'s – ekassis Sep 5 '16 at 11:03
• @ekassis - the 1st one-liner doesn't work on your solaris box. Use the second form (one subcommand per line, without comments...) or, if you prefer on one line use several expressions: `sed -e '/logical IO/{' -e 'x;//!{' -e 'g;p' -e '}' -e 'd' -e '}' -e '//!h' infile` – don_crissti Sep 5 '16 at 11:04
• the last command is working properly `sed -e '/logical IO/{' -e 'x;//!{' -e 'g;p' -e '}' -e 'd' -e '}' -e '//!h' infile` – ekassis Sep 5 '16 at 11:12
• @don_crissti, for Solaris `sed` (for anything but GNU sed essentially), you'd need to separate the `#` command from the preceding command (`x; # ...` for instance). – Stéphane Chazelas Sep 5 '16 at 11:13
• @ekassis, `sed -e '/logical IO/{x;//!{g;p;}' -e 'd;}' -e '//!h'` should be enough. Still not POSIX (yet, see also austingroupbugs.net/view.php?id=944) but portable. – Stéphane Chazelas Sep 5 '16 at 11:16

In the TXR language we can express this without any mutating state variables. At any given position in the file we can perform a multi-line pattern match with two branch alternatives: we either match one or more consecutive lines which contain the search string and then print the first, or else we match one line and print that. One possible way is this:

``````@(repeat)
@  (cases)
@    (collect :gap 0 :mintimes 1)
@line
@      (require (search-str line "logical IO"))
@    (end)
@    (do (put-line (first line)))
@  (or)
@line
@    (do (put-line line))
@  (end)
@(end)
``````

Run:

```\$ txr first-log-IO.txr data
select * from test1 where 1=1
testing logical IO 24
select * from test2 where condition=4
parsing logical IO 45
select * from test5 where 1=1
testing logical IO 24
select * from test5 where condition=78
parsing logical IO 346
```

`@(repeat)` establishes a walk through the data without collecting variable bindings; when this construct is seen it usually indicates that some side-effect takes place in the iteration. In this case it is output.

Inside the `@(repeat)` we have a `@(cases)` construct: multi-way match consisting of cases separated by `@(or)`. The second branch, the fallback case, of this is just `@line` which matches a line. The `@(do (put-line (first line)))` directive which follows prints that line.

The main branch of the `@(cases)` collects material via `@(collect)`. The matches must be consecutive, required by `:gap 0`, and there must be at least one, required by `:mintimes 1`. The collect body matches a single line, bound to the `line` variable. Then there is a `@(require ...)` assertion which fails unless the line contains the substring `"logical IO"`. The collection will therefore stop when it encounters a non-matching line, because the `:gap 0` prevents it from skipping it. The matching lines are implicitly collected into a list called `line` which pops out of the `collect` (a variable bound inside a collect automatically becomes a list outside of the collect, of all the values bound over the multiple iterations). We just print the first one, as required, suppressing the rest.

Note that the two `@line` matches have nothing to do with each other; they bind the `line` variable in different scopes.

Another way is to do some functional programming over lazy lists in TXR Lisp:

``````[(opip (partition-by (do cond
((search-str @1 "logical IO") t)
(t @1)))
(mapcar* first)
put-lines)
(get-lines)]
``````
\$ txr first-log-IO.tl

The `opip` operator is a syntactic sugar for constructing a pipeline of functions. Its arguments are all treated as `op` syntax: a macro for generating anonymous functions with implicit, numbered arguments.

The overall form is [(opip ...) (get-lines)] which just means "call the function produced by `opip`, with the result of `(get-lines)` as an argument". This `(get-lines)` converts the standard input stream into a lazy list of strings. (Its "opposite" is `put-lines`, which makes an appearance).

Now in the pipeline, we use `partition-by` to (lazily!) convert the list of lines into a list of lists which are its partitions. The partitioning condition is such that each line which contains `logical IO` is mapped to the symbol `t`, and all other lines just map to themselves. This means that consecutive lines which contain `logical IO` appear as a partition, and all other lines appear isolated as partitions of length one. All we have to do with this data now is map each partition to its first item, via `(mapcar* first)` and pass it to `put-lines` to dump the result.

We use `mapcar*` because that's the lazy version of `mapcar`. We want everything to be lazy so that the action is actually triggered by `put-lines`. As `put-lines` marches through the output list, it pulls items out of the lazy `mapcar*`, which forces the elements produced by the `partition-by`, which forces the list produced by `(get-lines)` which causes I/O to take place to read those lines.

If we use the regular `mapcar` by mistake, we cause the problem that the entire output is constructed in memory before being dumped out, which bodes badly for large files.