4

How do I compare a list of unsorted words of variable length and remove duplicate words that have a suffix?

Example word list:

    iron        
    curl
    curled
    railroad
    curler
    curling
    curls
    irons
    pan
    pans
    park
    parker
    parks
    parked
    railroads

Example Output:

    iron
    curl
    railroad
    pan
    park

The words all have different lengths, not just four or eight letters. I know how to search and print words containing suffixes but I'm not sure how to compare a list of words, some of which have suffixes, and then remove the words with suffixes, if there is a word that does not have a suffix on the list, without changing the sort order.

  • 2
    How do you define a duplicate word? Are pan and pander duplicates? Pan and panda? Pandemic and pandemonium? – Jeff Schaller Apr 21 '16 at 2:41
  • Thank you for replying. Pan, pander, panda, pandemic, and pandemonium are all different words that I would like to retain. However, pans, panders, pandas, and pandemics, I'd like to remove, if pan, pander, panda, and pandemic are on the list. Your example words, pan, panda, pander, and pandemic should all be on my list for the initial comparison. Pandemonium is not on my list. – J363 Apr 21 '16 at 3:23
  • 3
    Note that pans is not only the plural of pan. It's also a verb with a completely unrelated meaning to the noun (actually it's multiple verbs with multiple unrelated meanings) - e.g. pans out, the camera pans over the scene. So simply excluding words with s or ed etc suffixes will exclude too much. – cas Apr 21 '16 at 11:56
  • 1
    Why has pan disappeared between the input and the output? – Wildcard Apr 21 '16 at 21:47
  • 1
    @Wildcard it was ... panned ... – Jeff Schaller Apr 22 '16 at 0:14
7

You might need a word stemming algorithm for this. For example, Lingua::Stem is a word stemmer module written in Perl.

If this fits your needs, you would need to install Lingua::Stem via CPAN. Then, the following Perl script would do the job:

#!/usr/bin/perl

require Lingua::Stem;

# Read lines into array
chomp(my @words = <STDIN>);

# Stem in English
my $s = Lingua::Stem->new( -locale => 'en' );
my $stemmed = $s->stem_in_place( @words );

# Output result of stemmed words with duplicates removed
my $oldw = undef;
foreach $w (sort @$stemmed) {
    print "$w\n" unless ($w eq $oldw);
    $oldw = $w;
}

Example output:

$ ./stem.pl < inputfile
curl
curler
iron
pan
park
parker
railroad

Obviously, this deviates slightly from your example output due to the stemmer's interpretation of word suffixes which differs from yours in some cases. If this affects a moderate number of words in your application only, it is possible to define exceptions with the add_exceptions method:

...
$s->add_exceptions( { "parker" => "park", "curler" => "curl" } );
$stemmed = $s->stem_in_place( @words );
...
  • I appreciate your effort but when I use your proposed solution to process large text files it generates undesirable output. For example, above is trimmed to abov, abrasive becomes abrasiv, absence becomes absenc, etc. This is just a random sampling of a few lines from the list. The volume of errors is so great that the tool you have proposed is unusable. – J363 May 29 '16 at 21:48
0

This solution user "123" created for me on another question was able to strip suffixes reliably without mangling words. I wanted to come back and answer this question so that anyone seeking a similar solution could get a good answer.

awk 'FNR==NR{a[$0 "s"]++;next}!($0 in a)' file.txt file.txt
awk 'FNR==NR{a[$0 "ed"]++;next}!($0 in a)' file.txt file.txt
awk 'FNR==NR{a[$0 "ing"]++;next}!($0 in a)' file.txt file.txt
awk 'FNR==NR{a[$0 "ness"]++;next}!($0 in a)' file.txt file.txt
awk 'FNR==NR{a[$0 "er"]++;next}!($0 in a)' file.txt file.txt

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.