Removing text from pattern1 up to and including 2nd match of pattern2?

Question

I have a text file like so:

<!--START OF FILE -->
random text
<meta> more random text </meta>
x x x x x x x 
more random text
that I dont need 
x x x x x x x

I need everything
from this point
onwards
...

I need to remove everything between <!--START OF FILE --> and the second x x x x x x x like so:

I need everything
from this point
onwards
...

I tried using sed '/<!--START OF FILE -->/,/x x x x x x x/d' test.txt but this removes the block between the first occurence of x x x x x x x which is not what I want.

Answer 1

This is quite the opposite of

How to print lines between pattern1 and 2nd match of pattern2?

With sed you'd do something like:

sed -n '/PATTERN1/,$!{         # if not in this range
p;d                            # print and delete
}
/PATTERN2/!d                   # delete if it doesn't match PATTERN2
x;//!d                         # exchange and then, again, delete if no match
: do                           # label "do" (executed only after the 2nd match)
n;p                            # get the next line and print
b do' infile                   # go to label "do"

or, in one line (on gnu setups):

sed -n '/PATTERN1/,$!{p;d;};/PATTERN2/!d;x;//!d;: do;n;p;b do' infile

Sure, it's easier with awk and counters. I'll leave that as an exercise for you...

Answer 2

Straightforward awk:

$ awk '/<!--START OF FILE -->/ {a=2}; !a; /x x x x x x x/ && a {a--}' < data

I need everything
from this point
...

It just prints whenever a is zero and decrements it when it sees the x x x ....

Or starting from the actual start of the file instead of a pattern, change the first block to BEGIN {a=2}.

Note that your sample input has an empty line after the second x x x..., and it remains in the output if we stop removing lines at the x x x... line.

Answer 3

grep -Pz '(?s)<!--START OF FILE(.*?x x x x x x x){2}\K.*' input.txt

Explanation

1. grep -Pz

   • -P - Interpret the pattern as a Perl-compatible regular expression (PCRE).
   • -z - process the input.txt as one big line.

2. (?s)<!--START OF FILE(.*?x x x x x x x){2}\K.*

   • (?s) - Turn on "dot matches newline" for the remainder of the regular expression.
   • .*? - non-greedy matching.
   • {2} - amount of repetitions of the pattern.
   • \K - any previously-matched characters to be omitted from the final matched string.

Answer 4

This snippet:

# Utility functions: print-as-echo, print-line-with-visual-space.
pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }
pl " Input data file $FILE:"
head -v -n 20 $FILE

pl " Expected output on file $E:"
head -v $E

pl " Results:"
cgrep -V -D -w '<!--START OF FILE -->' +2 +w 'x x x x x x x' 'meta' $FILE

produces:

-----
 Input data file data1:
==> data1 <==
<!--START OF FILE -->
random text
<meta> more random text </meta>
x x x x x x x 
more random text
that I dont need 
x x x x x x x

I need everything
from this point

-----
 Expected output on file expected-output1:

I need everything
from this point
onwards
...

-----
 Results:

I need everything
from this point
onwards
...

This omits (-V) a window beginning (-w) with '...START...', and ending (+w) with the second occurrence (+2) of a string '...x x...' that has the string 'meta' inside the window.

On a system like:

OS, ker|rel, machine: Linux, 3.16.0-4-amd64, x86_64
Distribution        : Debian 8.9 (jessie) 
bash GNU bash 4.3.30

Some details for cgrep:

cgrep   shows context of matching patterns found in files (man)
Path    : ~/executable/cgrep
Version : 8.15
Type    : ELF 64-bit LSB executable, x86-64, version 1 (SYS ...)
Home    : http://sourceforge.net/projects/cgrep/ (doc)

Although one would need to get and compile cgrep, I have had no trouble doing that on 32-bit or 64-bit systems, and it is available on macOS (High Sierra) with brew. The execution time is on a par with GNU grep.

Best wishes ... cheers, drl