Aside from the problem you're asking about, your script has a massive flaw in that it makes one complete pass through 'foo' for every line in 'remove.txt'. This is extremely ineffficient. The better way to do it is to read in 'remove.txt', construct a single long regular expression, and then use that once to edit 'foo'.
The simplest way to do that is to push the search strings into an array and then 'join()' the array with a '|' (regexp "or") character to create a string which can be used as a regular expression.
Here's a script that does that and fixes your original problem.
# first construct a regular expression containing every
# line that needs to be removed. This is so we only have
# to run a single pass through $infile rather than one
# pass per line in $removefile.
my @remove = ();
open(REMFILE,"<",$removefile) || die "couldn't open $removefile: $!\n";
next if (/^\s*$/);
push @remove, $_;
# choose one of the following two lines depending on
# whether you want to remove only entire lines or text
# within a line:
my $remove = '^(' . join("|",@remove) . ')$';
#my $remove = join("|",@remove);
# now remove the unwanted text from all lines in $infile
my $infile = 'foo';
# if you want to delete matching lines, try this instead:
#system('perl','-n','-i','-e',"print unless /$remove/",$infile);