3

I've a file with this kind of content :

bla bla
pattern2
bla
pattern1
pattern2
bla
bla pattern1 bla
bla
pattern1

I would like to remove the whole line in bold, ie contains first occurrence of pattern1 after last matched of pattern2.

Does someone have an idea ?

Thanks !

5

This is an ex one-liner. (ex is the predecessor and scripted form of vi.)

printf '%s\n' '$?pattern2?/pattern1/d' x | ex file.txt

The x saves and exits. Change it to %p if you want to just print the altered file but not save changes (good for testing).

$ means last line of file; ?pattern2? is an address meaning the first result of a backwards search for pattern2 starting from the current position; /pattern1/ is a forward-searching address, and d is the line deletion command.

Use ex when you need forward AND backward addressing.


You can do the same thing interactively in vi or Vim:

vim file.txt

Then, type

:$?pattern2?/pattern1/d

and press Enter.

Then save and exit with :x Enter.

  • 1
    The ex script would not work in case there were dangling pattern2 lines in the trailing portions of the input followed by some more lines. – user218374 Aug 12 '17 at 15:12
  • @RakeshSharma good point! Likely also if pattern2 is on the final line. Like to suggest an edit? :) – Wildcard Aug 12 '17 at 17:10
  • No there's no need for it, as the ex soln. is well within the OP's spec(I think). My earlier comment was just to let know the corner cases where it might not work. – user218374 Aug 12 '17 at 17:13
0

Here is a brute force method. Reads the data and loops through it twice. Finds the last occurrence of pattern2 the first time through and the first occurence of pattern1 the second time.

#!/usr/bin/perl

# usage:  perl remove-pattern.pl [file]
use strict;

# reads the contents of the text file completely
# removes end of line character and spurious control-M's
sub load {
   my $file = shift;
   open my $in, "<", $file or die "unable to open $file : $!";
   my @file_contents = <$in>;
   foreach ( @file_contents ) { 
      chomp; 
      s/\cM//g; 
   }
   return @file_contents;
}

#  gets the first file from the command line
#  after the perl script
my $ifile = shift;

# read the text file
my @file_contents = &load($ifile);

# set 2 variables for the index into the array 
my $p2 = -1;
my $p1 = -1;

# loop through the file contents and find the last
# of pattern2 (could go reverse the data and find the 
# first of pattern2
for( my $i = 0;$i < @file_contents; ++$i ) {
   if( $file_contents[$i] =~ /pattern2/) {
      $p2 = $i 
   } 
}

# start at the location of the last of pattern2
# and find the first of pattern1
for( my $i = $p2; $i < @file_contents; ++$i ) {
   if($file_contents[$i] =~ /pattern1/) {
     $p1 = $i ;
     last;
   }
}

# create an output file name
my $ofile = $ifile . ".filtered";

# open the output file for writing
open my $out, ">", $ofile or die "unable to open $ofile : $!"; 

# loop through the file contents and don't print the index if it matches
# p1.  print all others
for( my $i = 0;$i < @file_contents; ++$i ) {
   print $out "$file_contents[$i]\n" if ($i != $p1);
}


--- data.txt  ---
bla bla
pattern2
bla
pattern1
pattern2
bla
bla pattern1 bla
bla
pattern1

If the above perl script were named 'remove-pattern.pl', it would be executed with the following command given the data.txt input file. %> perl remove-pattern.pl data.txt

Resulting output file 'data.txt.filtered'

--- data.txt.filtered ---
bla bla
pattern2
bla
pattern1
pattern2
bla
bla
pattern1
0

To find the line number of that line:

lineno=$( nl file | tac | awk '/pattern1/ {last = $1} /pattern2/ {print last; exit}' )

Using nl to add line numbers to the file,
tac to reverse the lines,
and awk to print the line number for the last "pattern1" before the first "pattern2".

And then to delete that line:

sed -i "${lineno}d" file
0

I have no computer here to test, but this should work with gnu sed:

sed 'H;1h;$!d;g;s/.*pattern1/@@@/;s/\n[^\n]*pattern2[^\n]*//;H;g;s/\(.*pattern1\).*@@@/\1/'

Instead of @@@ use any sequence of characters that is known to be not part of the file.

0

If you wanted to make only one pass in the file and minimise the number of lines to hold in memory, you could use awk with a state machine approach. Those don't make for the shortest solutions but are easy to come up with and read/maintain. You can replace the state names with numbers to make it (possibly) more efficient.

PATTERN1=pattern1 PATTERN2=pattern2 awk '
  BEGIN {
    p1 = ENVIRON["PATTERN1"]
    p2 = ENVIRON["PATTERN2"]
    state = "init"
  }
  state == "init" {
    if ($0 ~ p2) state = "p2_found"
    print
    next
  }
  state == "p2_found" {
    if ($0 ~ p1) {
      state = "p1_found"
      p1_line = $0
      printf "%s", hold
      hold = ""
    } else if ($0 ~ p2) {
      # we can print the text held since the last p2
      printf "%s", hold
      hold = $0 RS
    } else hold = hold $0 RS
    next
  }
  state == "p1_found" {
    if ($0 ~ p2) {
      state = "p2_found"
      # the line that matched p1 is not discarded
      printf "%s\n%s", p1_line, hold;
      hold = ""
    }
    hold = hold $0 RS
  }
  END {
    # here we are not printing p1_line which is how it is discarded
    printf "%s", hold
  }'

(I'm assuming there are no lines that match both pattern1 and pattern2).

0

We can preform the check using sed, although the code below is written for GNU sed, where we maintain the last full range /pat2/../pat1/ in the hold space. And, then we maintain an envelop of expanding frontier starting from the line next to the last line of the range stored in hold and advancing towards the eof.

sed -e '
   /pattern2/,/pattern1/!b
   H;/pattern2/h;/pattern1/!{$!d;g;q;}
   ${g;bc;}
   N;s/.*\n//
   :a
      $bd;N
   /pattern2/!ba
   :b
      $bd;N
   /pattern2.*\n.*pattern1/!bb
   x;$!{n;/pattern2/bb;ba;}
   G
   :c;s/\(.*\)\n.*/\1/;q
   :d;x;s/\(.*\)\n.*/\1/;G
' input,txt

In this method we slurp the whole file into the pattern space, then using the power of regexes find and determine the last /pattern1/ line in a range and delete it.

sed -Ee '
   $!{N;H;s/.*//;x;D;}
   /pattern2/!q;/pattern1/!q;/pattern2.*\n.*pattern1/!q
   h;s/((.*\n)?[^\n]*pattern2[^\n]*)\n(.*pattern1.*)/\3/
   s/^//;tdummy
   :dummy
   s/\n[^\n]*pattern1[^\n]*\n/\n/;ta
   s/^[^\n]*pattern1[^\n]*\n//;ta
   s/\n[^\n]*pattern1[^\n]*$//;ta
   s/^[^\n]*pattern1[^\n]*$//
   :a;x;s/((.*\n)?[^\n]*pattern2[^\n]*)\n.*pattern1.*/\1/;G
' input.file
0

Sed:

n=$(sed -ne '/pattern2/,/pattern1/{/pattern1/=;}' yourfile | tail -n 1)
sed -i'' -e "${n}d" yourfile

Sed: based on 1-pass

# invoke GNU sed with extended RE(-E), slurp mode(-z), in-place editing(-i) options
sed -i -Eze '
   h;s/(.*pattern2[^\n]*)\n.*/\1/p;   # traverse till the last pat2 line and print it
   g;s/.*pattern2[^\n]*(\n.*)/\1/;    # remove till the last pat2 line
   s/\n[^\n]*pattern1[^\n]*//;        # now look for the 1st occurrence of pat1
   ;                                  # clip that line, & print what remains
' inp

Working:

  • Determine the line number of the /pattern1/ but this should lie within the proper range only, i.e., /pattern2/,/pattern1/ for it to be considered.
  • Take the last of these numbers.
  • In the 2nd pass, supply the line number determined above and delete that line.

Perl:

     |----A---|   |----B----|--------C--------|D|----------E---------|-F-|
perl -0777pi -e 's/.*pattern2(?:(?!pattern1).)*\K(?-s:\n.*pattern1.*$)//ms' yourfile
  • A: Invoke Perl in slurp mode(-0777) + line-by-line read-in + autoprint(-p) enabled, in-place editing(-i)
  • B+C: Traverse upto the last occurrence of pattern2 then slow down and go steadily and tidying over non pattern1
  • D: When you reach here, that is the last pit stop where pattern1 was not seen after the last pattern2, mark it with a \K. meaning this will not appear in the consumed part of input rather just the matched part.
  • E: This we turn OFF the /s cloistered pattern match modifier, meaning, in this portion of the match, the . no longer matches newline, meaning we can't jump past lines. We just match the whole single line which has pattern1 and we remove this. What remains in the pattern space is autoprinted.
  • -F: We invoke the s/// with /s and /m modifiers. We will selectively turn off the /s modifier inside the regex to tailor to our needs.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.