This is very similar to @meuh's answer, but instead of adding a foreach
loop after the split
line, you only need to add one line there, using either perl
's grep
function or its map
function:
@titlewords = grep (!/^(and|if|the)$/i, @titlewords);
or
@titlewords = map { /^(and|if|the)$/i ? () : $_ } @titlewords;
See perldoc -f grep
and perldoc -f map
for more details on these functions and the differences between them. They are commonly used (especially map
) in many perl
scripts, so it is worth taking the time to understand what they do and learn how they work.
BTW, do NOT use #!/usr/bin/env perl
. Using env
like that is bad enough with python
and ruby
scripts (where it is, unfortunately, the convention), but it is completely broken for perl
scripts and definitely NOT the conventional way to run them.
perl
has many command-line options which alter its behaviour in significant ways, depending on what kind of program you're trying to write. Using env
to run an interpreter like perl
completely destroys the ability to pass command-line options to the interpreter (because env
does not support it. env
wasn't even designed to be used for this purpose, doing so is just an ugly hack that takes advantage of a side-effect of env
's actual purpose - which is setting environment variables before running a program).
Use #!/usr/bin/perl
instead (or whatever the path to your perl
interpreter is).
Here's another perl script that does what you want - but this one uses the Class::CSV
and List::Compare
modules, as well as two Hashes-of-Arrays to compare the CSV files:
#! /usr/bin/perl
use strict;
use warnings;
use Class::CSV;
use List::Compare;
sub parse_csv($%) {
my($filename,$tw) = @_;
# exclude the following word list and the "empty word"
my @exceptions = qw(and if the);
my $exceptions = '^(\s*|' . join('|',@exceptions) . ')$';
my $csv = Class::CSV->parse(
filename => $filename,
fields => [qw/id title num1 num2/]
);
# build a hash-of-arrays (HoA), keyed by the CSV line. Each array
# contains the individual words from each title for that line (except
# for those matching $exceptions). The words are all converted to
# lowercase to enable case-insensitive matches.
foreach my $line (@{$csv->lines()}) {
# The following three lines are required because the input file has
# fields separated by ', ' rather than just ',' which makes
# Class::CSV interpret the numeric fields as strings.
# It's easier/quicker to do this than to rewrite using Text::CSV.
#
# The final output will be properly-formed CSV, with only a comma as
# field separator and quotes around the title string.
my $key = join(',',$line->id,'"'.$line->title.'"',$line->num1,$line->num2);
$key =~ s/([",])\s+/$1/g; # trim whitespace immediately following " or ,
$key =~ s/\s+([",])/$1/g; # trim whitespace immediately preceding " or ,
# If it wasn't for the not-quite-right CSV format, we could just use:
#my $key = $line->string;
push @{ $tw->{$key} }, grep (!/$exceptions/oi, split(/\s+/,$line->title));
};
};
# two hashes to hold the titlewords HoAs
my %tw1=();
my %tw2=();
parse_csv('csv1',\%tw1);
parse_csv('csv2',\%tw2);
# now compare the HoAs
foreach my $k2 (sort keys %tw2) {
my @matches = ();
foreach my $k1 (sort keys %tw1) {
my $lc = List::Compare->new('-u', \@{ $tw2{$k2} }, \@{ $tw1{$k1} });
push @matches, $k1 if ($lc->get_intersection ge 3);
};
print join("\n",sort(@matches,$k2)),"\n\n" if (@matches);
};
Output:
11,"The Sun Still Shines in Reading",64312,464566
97,"Reading Still Shines",545464,16748967
Each group of matches is sorted and, even though the sample output doesn't show it (because there's only one group of matches), each group is printed as a separate paragraph (i.e. separated by a blank line)
BTW, if you don't want the double-quotes around the title fields, edit the my $key=join(...)
line that adds them so that it doesn't.
perl
script in my answer below also answers your recently deleted question. Theget_intersection
method from theList::Compare
module eliminates duplicate entries (which is what you'd expect from set intersection), individual words are counted only once for each title. This is one of the huge benefits of using existing library modules rather than writing your own - they usually include useful features that would take a lot of effort to implement yourself. They also tend to have solved most or all of the common (and many not-so-common) problems related to the task at hand.