10

I have a text file, and I have a pattern that I want grep not to match. Thing is, I also want the line before not to match.

My file:

line 1
line 2
pattern
line 4

And I tried cat file | grep -v pattern, which outputs:

line 1
line 2
line 4

Then I tried cat file | grep -B 1 pattern, which outputs:

line 2
pattern

However, when I use both of them together cat file | grep -v -B 1 pattern, I get:

line 2

How can I make it so that the output is:

line 1
line 4

7 Answers 7

8

I tend only to use grep when extracting single lines from files, so when I need to perform more complicated edits in a text, I use other tools.

All solutions here assume that the pattern may occur multiple times in the text and will remove the lines on which it occurs and the lines immediately previous to them. The first two solutions will have issues if the pattern matches on consecutive lines.


You can use sed to match a pattern with /pattern/ and let that trigger the commands N and d, which appends the next line to the buffer and then discards both:

sed '/pattern/ { N; d; }' file

Since you want to discard the line before the match of the pattern, we feed the data backwards into sed, starting with the last line and moving towards the start of the file. Then we reverse the data again when sed is done.

tac file | sed '/pattern/ { N; d; }' | tac

The tac utility is part of GNU coreutils. Most non-GNU systems may use tail -r in place of tac (check your tail(1) manual).

If the pattern matches two consecutive lines, this will fail to remove the line previous to the first of those lines (since the first line would get deleted).


Using the ed editor:

printf '%s\n' 'g/pattern/ -1,. d' ,p Q | ed -s file

This applies the command g/pattern/ -1,. d to the contents of the file. This command searches for each line that matches pattern, and then deletes that line and the line previous to it.

The final ,p and Q editing command prints the whole file and quit the editor without saving.

If the pattern matches two consecutive lines, this will remove the line that becomes previous to the second line after removing the line previous to the first line.

(That last sentence was correct when I wrote it, but it's obviously a write-only sentence.)


We can also use grep and its non-standard but commonly implemented -B option for giving us the line numbers that need to be deleted. These numbers can be converted to a sed script that we run on the original data:

grep -n -B1 'pattern' file | sed 's/[:-].*/d/' | sed -f /dev/stdin file

The grep command would, given the text in the question, output

2-line 2
3:pattern

... and the first sed command converts this into the sed editing command 2d followed by 3d ("delete line 2 and 3"). The last sed command in the pipeline takes this editing script and applies it to the original text.

This variant has no issues with consecutive lines matching the pattern as it uses a kind of 2-pass approach, first finding all lines that should be deleted and then deleting them (instead of deleting lines while reading the text for the first time).

0
5

Using any awk with tac, you can delete any number of lines before a matching pattern:

$ tac file | awk '/pattern/{c=2} !(c&&c--)' file | tac
line 2
line 1

Just change c=2 to c=5 or whatever number of lines you want to delete up to and including the matching line, e.g. to delete a line containing the number 97 and the 94 lines before it:

$ seq 100 | tac | awk '/97/{c=95} !(c&&c--)' | tac
1
2
98
99
100

Now try doing that with sed instead of awk :-).

See printing-with-sed-or-awk-a-line-following-a-matching-pattern for an explanation of this and other related idioms.

2
  • 2
    awk, an elegant weapon from a more civilized age... Oct 7, 2022 at 9:13
  • @JoL no they are not the same. && is a short-circuit operator, the second operand will come into play if and only if the first one is Truthy
    – Sundeep
    Oct 7, 2022 at 9:51
1

Note: This code only works if in the file does not exist duplicate lines or substrings of every line which match with the output of grep -B1 pattern file.

For example, if file contains the following lines:

line 1
line 2
line 2
pattern
line 1 line 2
line 3

And I use grep -B1 pattern file | grep -v "$(cat)" file the output will not be as you expected:

line 1
line 3

The best way to solve that is using Kusalananda's answer

Solution (this one only applies to cases where there are no duplicate lines or substrings as I explained above)

In bash this works for me (I think there are better ways):

grep -B1 pattern file | grep -v "$(cat)" file

In zsh the commands above will not work. I don't know why. But you can use:

grep -B1 pattern file | { val="$(cat)" ; grep -v "$val" file; }

P.S. You don't have to use cat your_file | grep pattern that's redundant. You should use grep pattern your_file

5
  • 1
    See the -f option to grep, and you will have to cover the case where there are duplicate lines (change all the numbers in the example to 2), as well as lines that are substrings of other lines. Additionally, the commands that you show will deadlock in the zsh shell.
    – Kusalananda
    Oct 6, 2022 at 19:03
  • @Kusalananda oh yeah! I did not thing in that case! My bad! About zsh only the first command will deadlock. Oct 6, 2022 at 19:15
  • @Kusalananda is it possible to solve that problem with grep -f? I'm not sure, I thing is not possible with grep at all Oct 6, 2022 at 19:21
  • No, it's not. The newline-separated patterns given to grep, either on the command line or via -f, can match anywhere in the text separately from each other. grep will not match them together as one pattern.
    – Kusalananda
    Oct 6, 2022 at 19:28
  • @Kusalananda thanks! I really was thinking that. I'll update the answer to detail the problems. Oct 6, 2022 at 19:32
1

The solutions by Kusalananda and Ed Morton are the simplest and most practical, but they require either reading the contents twice or reading the contents whole before starting. Pipes can't be re-read nor are they always finite. A solution that works with any text stream can be something like this:

$ awk -v pat='^pattern$' '
  function set_prev() { prev_present = 1; prev = $0 }
  NR == 1 { set_prev() }
  $0 ~ pat { prev_present = 0; next }
  NR != 1 { if (prev_present) print prev; set_prev() }
  END { if ($0 !~ pat) print }
' << EOF
pattern
line 1
line 2
pattern
pattern
line 4
line 5
pattern
EOF
line 1
line 4

If you want to make the number of lines not printed to be variable, that gets a bit more complicated:

$ awk -v n=2 -v pat='^pattern$' '
  function ring_empty() { i = 0; ring_is_full = 0 }
  function ring_is_empty() { return i == 0 && !ring_is_full }
  function ring_add() { ring[i++%n] = $0; ring_is_full = i >= n }
  function ring_starting_index() { return ring_is_full ? i%n : 0 }
  function ring_print_oldest() { print ring[ring_starting_index()] }
  function ring_print_all() {
    if (ring_is_empty()) return
    j = ring_starting_index()
    do {
      print ring[j%n]
    } while (++j%n != i%n)
  }
  function ring_push_out() {
    if (ring_is_full) ring_print_oldest()
    ring_add()
  }

  { ring_push_out() }
  $0 ~ pat { ring_empty() }
  END { ring_print_all() }
' << EOF
pattern
line 1
line 2
pattern
pattern
line 4
line 5
pattern
EOF
line 1
line 4

Neither of these solutions have problems with consecutive matches, btw.

1

You could use pcregrep and its Multiline mode:

pcregrep -Mv '\n.*pattern'

Note that if the first line matches the pattern, it won't be removed. That can be worked around by using:

pcregrep -Mv '(\n)?.*pattern'

(the (...) around \n apparently necessary, I don't know why it doesn't work with \n?.*pattern or [\n]?.*pattern with version 8.39 here).

4
  • Best answer, IMO. Simplest and no problems with consecutive matches, nor with matches at the beginning or end if done like this: pcregrep -Mv '(.*\n){0,1}pattern'. That also simplifies extending this to remove more lines by changing the 1.
    – JoL
    Oct 10, 2022 at 8:39
  • @JoL. Thanks. I tried with \n? but it doesn't work. You made me realise it works with (\n)?. See edit. Oct 10, 2022 at 8:53
  • \n? is working with me. My version is 8.45. I guess it's a bug that got fixed.
    – JoL
    Oct 10, 2022 at 9:00
  • 1
    Interestingly fails to remove previous lines when they're input by hand or pasted into the terminal. I guess multiline mode simply doesn't work with tty input.
    – JoL
    Oct 10, 2022 at 9:27
0

Using Raku (formerly known as Perl_6)

raku -e 'for lines.join("\n") { print .subst(:global, / \N* \n pattern [ \n | $ ]/) };'     

#OR

raku -e 'for lines.join("\n") { print .split(/ \N* \n pattern [ \n | $ ] /).join };'      

These first two answers (above) basically detect a two-line pattern, and delete it. Thus consecutive occurrences of the word pattern are not handled, nor is the occurrence of pattern in the first line. For both answers, lines are read-in from a file and joined back together on \n newlines (since lines autochomps by default). Then the desired two-line regex is searched for and 1). substituted (with nothing, i.e. deleted) or 2). split on the two-line regex and joined the output.

The next two answers (below) handle the occurrence of pattern in the first line, as well as handle consecutive occurrences of the word pattern. They use the grouping [\N* \n]? at the start of the Regex:

raku -e 'for lines.join("\n") { print .subst(:global, / [\N* \n]?  pattern [ \n | $ ]/) };'     

#OR

raku -e 'for lines.join("\n") { print .split(/ [\N* \n]?  pattern [ \n | $ ] /).join };'      

Sample Input:

pattern
line 1
line 2
pattern
pattern
line 5
line 6
pattern

Sample Output (first 2 examples that delete a 2-line Regex):

pattern
line 1
pattern
line 5

Sample Output (examples 3 and 4 that additionally handles pattern in first line as well as consecutive occurrences of pattern):

line 1
line 5

FYI: Raku's lines routine is advertised to act lazily, so potentially a file can be analyzed without having to read-in the entire file first. For commentary on Raku's lines routine, see the URL below.

https://speakerdeck.com/util/reading-files-cant-be-this-simple
https://raku.org

Special thanks to user @JoL for an insightful critique of the original regexes in this answer.

4
  • 1
    For pattern\nline 1\nline 2\npattern\npattern\nline 4\npattern\n, fails to remove the first, third, and fourth match. The first and third can be fixed with (\N*\n)?. Regarding the fourth, it's because you're using the newline as a line separator instead of a line terminator. You should append a newline on each then join with empty string. Not sure what the syntax is for that.
    – JoL
    Oct 8, 2022 at 2:47
  • 1
    (\n|$) also works, but you need to change the .put for .print to avoid an extra newline added to that of line 1\n.
    – JoL
    Oct 8, 2022 at 2:54
  • Regarding lines being lazy, the whole thing isn't working lazily. You can test by just running it and inputting the lines by hand. It won't give results until you ^D at the end. It's probably the case that while lines may be lazy, join isn't and is pulling to whole file to build a string. This is my first time touching Raku, though, so I can't say for certain.
    – JoL
    Oct 8, 2022 at 3:00
  • Thank you for the exceptional feedback! Answer has been amended. Oct 8, 2022 at 4:43
-2

You can store the line containing pattern and the line above that in a variable. Then you can use this variable to grep again in your file.

match=$(grep pattern -B1 file)
grep "$match" -v file
2
  • 2
    Test this with all numbers in the example changed to the same number.
    – Kusalananda
    Oct 6, 2022 at 19:05
  • 1
    See also comments under Edgar's answer.
    – Kusalananda
    Oct 6, 2022 at 19:29

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .