I have a file ~/foo and another file ~/remove.txt. I would like to write a perl script that loops over every line of ~/remove.txt and removes every instance of the string (<LINE>) from ~/foo (where <LINE> is the line in ~/remove.txt). My code is

#! /usr/bin/perl 

use strict;
use warnings;

sub main
    my $infile  = "remove.txt";

    open(INPUT, $infile) or die "cannot open $infile";

    while(my $line = <INPUT>)

    my $bad_string = "($line)";

    system( q( perl -p -i -e 's/$bad_string//g' foo ) );




But running this script does not seem to alter my file at all. What's wrong with my script?

  • What if a $bad_string contains a special character like a ??
    – sid_com
    Sep 20 '15 at 9:21
  • Why are you making a system call to perl, from within perl?
    – Sobrique
    Sep 21 '15 at 15:25
  • Because my skills are very limited and this is what I thought to do... Sep 21 '15 at 15:55

Aside from the problem you're asking about, your script has a massive flaw in that it makes one complete pass through 'foo' for every line in 'remove.txt'. This is extremely ineffficient. The better way to do it is to read in 'remove.txt', construct a single long regular expression, and then use that once to edit 'foo'.

The simplest way to do that is to push the search strings into an array and then 'join()' the array with a '|' (regexp "or") character to create a string which can be used as a regular expression.

Here's a script that does that and fixes your original problem.

#! /usr/bin/perl 

use strict;
use warnings;

# first construct a regular expression containing every
# line that needs to be removed.  This is so we only have
# to run a single pass through $infile rather than one
# pass per line in $removefile.
my @remove = ();

my $removefile='remove.txt';
open(REMFILE,"<",$removefile) || die "couldn't open $removefile: $!\n";
while(<REMFILE>) {
    next if (/^\s*$/);
    push @remove, $_;

# choose one of the following two lines depending on
# whether you want to remove only entire lines or text
# within a line:
my $remove = '^(' . join("|",@remove) . ')$';
#my $remove = join("|",@remove);

# now remove the unwanted text from all lines in $infile
my $infile = 'foo';

# if you want to delete matching lines, try this instead:
#system('perl','-n','-i','-e',"print unless /$remove/",$infile);
  • BTW, if you want to be able to have (and ignore) '#' comments in 'remove.txt', add s/#.*//; immediately after the chomp; line.
    – cas
    Sep 20 '15 at 2:26
  • Ahh... This was what I set out to do but honestly I don't have enough perl knowledge to write this myself. This really helps me learn so thanks! Sep 20 '15 at 2:27
  • that's why i wrote it - glad you learned something useful. BTW, just like your original script, the lines in 'remove.txt' are treated as regular expressions not as literal text....so you have to be careful if they contain regexp special characters like | or * or . etc. use \ to escape them in 'remove.txt' if you want them to be literals.
    – cas
    Sep 20 '15 at 2:34

You need to use qq() and escape regex meta characters (( and )) in $bad_string.

            my $bad_string = "\\($line\\)";
            system( qq( perl -p -i -e 's/$bad_string//g' foo ) );
  • I mad an error in my post. I have the line my $bad_string = "($line)"; right after chomp($line);. After using qq would this work? Sep 20 '15 at 2:00
  • @BrianFitzpatrick Ok, updated.
    – yaegashi
    Sep 20 '15 at 2:05

There's 3 elements to your problem:

  • building an 'exclude list' - note that 'special' characters in your exclusion list can cause problems.
  • reading your file, excluding a line if it 'matches'.
  • writing your new file.

In your question - I think there's a couple of things I'd call 'bad style'.

  • lexical filehandles with 3 argument open are good style.
  • calling system to run perl from within perl is inefficient.
  • quote interpolation is a nuisance best avoided
  • you're re-processing your output file repeatedly, which is horribly inefficient. (Remember - disk IO is the slowest thing you'll do on your system).

So with that in mind - here's how I'd do it:

#!/usr/bin/env perl
use strict;
use warnings;

my $infile = "remove.txt";
open( my $pattern_fh, '<', $infile ) or die "cannot open $infile $!";

#quotemeta escapes meta characters that'll break your pattern matching. 
my $regex = join( '|', map {quotemeta} <$pattern_fh> );
#compile the regex
$regex = qr/^($regex)$/;    #whole lines

print "Using regular expression: $regex\n"; 

open( my $input_fh,  '<', "foo" )     or die $!;
open( my $output_fh, '>', "foo.new" ) or die $!;

#tell print where to print by default. 
#could instead print {$output_fh} $_; 
while (<$input_fh>) {
    print unless m/$regex/;

#rename/copy if it worked

(NB: Not tested exhaustively - if you can give some sample data, I'll test/update as needed)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.