4

There are duplicates in my booklist (txt file) like the following -

The Ideal Team Player
The Ideal Team Player: How to Recognize and Cultivate The Three Essential Virtues
Ideal Team Player: Recognize and Cultivate The Three Essential Virtues

Joy on Demand: The Art of Discovering the Happiness Within
Crucial Conversations Tools for Talking When Stakes Are High
Joy on Demand

Search Inside Yourself: The Unexpected Path to Achieving Success, Happiness
Search Inside Yourself
......
......
......

I need to find the duplicate books and manually remove them after checking. I searched and found the lines need a pattern.

Ex.

Remove duplicate lines based on a partial line comparison

Find partial duplicate lines in a file and count how many time each line was duplicated?

But in my case finding a pattern in lines is difficult. However, I found a pattern in the sequence of words.

I want to mark lines as duplicate only if they have three consecutive words (case insensitive).

If you see you will find that in -

The Ideal Team Player
The Ideal Team Player: How to Recognize and Cultivate The Three Essential Virtues
Ideal Team Player: Recognize and Cultivate The Three Essential Virtues

Ideal Team Player is the consecutive words which I am looking for.

I would like the output to be something like the following -

3 Ideal Team Player
2 Joy on Demand
2 Search Inside Yourself
......
......
......

How can I do that?

7

The following awk program stores a count for how many times each set of three consecutive words occurs (after removing punctuation characters), and prints the counts and the set of words at the end if the count is larger than 1:

{
        gsub("[[:punct:]]", "")

        for (i = 3; i <= NF; ++i)
                w[$(i-2),$(i-1),$i]++
}
END {
        for (key in w) {
                count = w[key]
                if (count > 1) {
                        gsub(SUBSEP," ",key)
                        print count, key
                }
        }
}

Given the text in your question, this produces

2 Search Inside Yourself
2 Cultivate The Three
2 The Three Essential
2 Joy on Demand
2 Recognize and Cultivate
2 Three Essential Virtues
2 and Cultivate The
2 The Ideal Team
3 Ideal Team Player

As you can see, this may not be so useful.

Instead, we can collect the same count information and then do a second pass over the file, printing each line that contains a word triplet with a count larger than one:

NR == FNR {
        gsub("[[:punct:]]", "")

        for (i = 3; i <= NF; ++i)
                w[$(i-2),$(i-1),$i]++

        next
}

{
        orig = $0
        gsub("[[:punct:]]", "")

        for (i = 3; i <= NF; ++i)
                if (w[$(i-2),$(i-1),$i] > 1) {
                        print orig
                        next
                }
}

Testing on your file:

$ cat file
The Ideal Team Player
The Ideal Team Player: How to Recognize and Cultivate The Three Essential Virtues
Ideal Team Player: Recognize and Cultivate The Three Essential Virtues

Joy on Demand: The Art of Discovering the Happiness Within
Crucial Conversations Tools for Talking When Stakes Are High
Joy on Demand

Search Inside Yourself: The Unexpected Path to Achieving Success, Happiness
Search Inside Yourself
$ awk -f script.awk file file
The Ideal Team Player
The Ideal Team Player: How to Recognize and Cultivate The Three Essential Virtues
Ideal Team Player: Recognize and Cultivate The Three Essential Virtues
Joy on Demand: The Art of Discovering the Happiness Within
Joy on Demand
Search Inside Yourself: The Unexpected Path to Achieving Success, Happiness
Search Inside Yourself

Caveat: This awk program needs enough memory to store the text of your file about three times over, and may find duplicates in common phrases even when the entries are actually not truly duplicated (e.g. "how to cook" may be part of the titles of several books).

  • Is this script case insensitive? And in there any way if two lines match then do not search for further matches? For example, when Cultivate The Three matched between two lines, I do not need to match The Three Essential or Three Essential Virtues anymore? – blueray Aug 28 at 7:12
  • I mean, after the first match is found between two likes, skip to next like. I do not need every match between two lines but the first match. This way the result will be smaller and easy to crosscheck manually. – blueray Aug 28 at 7:15
  • 1
    @blueray Making it case insensitive would be fairly easy, but the the second tweak would require remembering which word triplets "belong together". That would involve a bit more bookkeeping, and I'm unable to look into it at this exact moment (but maybe later, after work). – Kusalananda Aug 28 at 7:16
  • Thank you very much. Please remember to look into it. – blueray Aug 28 at 7:17
  • 1
    @blueray Sorry for not getting back to this (busy week). No, the two awk scripts in my answer are two totally independent solutions and were never supposed to be stringed together like that. The second script implements a two-pass algorithm, and therefore has to read the file twice. This is why I run it with your input file mentioned twice on the command line ("file file"). – Kusalananda Sep 1 at 7:11
3

IMO, this task is better solved using the intersections of sets of words, rather than looking for 3 consecutive words.

Accordingly, the following perl script does not look for 3 consecutive words. Instead, it first reads in the entire input (from stdin and/or one or more files) and (using the Set::Tiny module) creates a set of words for each input line.

Then it processes the input a second time, and (for each line) it prints out any lines read in the first pass which have exact duplicates or where the intersection of sets has 3 or more elements.

It uses a hash array called %sets to store the word sets for each title, and another hash called %titles to count the number of times it has seen each title - this is used in the output phase to ensure it never prints any title more often than it was seen in the input.

In short, it prints duplicate lines and similar lines (i.e. those which have at least 3 of the same words in them) next to each other - the 3 words do not have to be consecutive.

The script ignores several very common small words when constructing the sets, but this can be disabled by commenting out or deleting the line with the OPTIONAL... comment. Or you can edit the common word list to suit your needs.

One thing worth mentioning is that the small words list in the script includes the word by. You can delete it from the list if you like, but the reason why it's there is to stop the script from matching on by plus any two other words - e.g. Aardvark Taxidermy for Personal Wealth by Peter Smith would match The Wealth of Nations by Adam Smith (matches on by, Wealth, and Smith). The first book is (I hope) entirely non-existent but if it did exist, it would not be at all related to an economics text.

Note: this script stores the entire input, and the associated word sets for each input line, in memory. This is unlikely to be a problem for modern systems with a few GiB of free RAM unless the input is extremely large.

Note2: Set::Tiny is packaged for Debian as libset-tiny-perl. It may be available pre-packaged for other distributions too. Otherwise, you can get it from the CPAN link above.

#!/usr/bin/perl -w

use strict;
use Set::Tiny;

# a partial list of common articles, prepositions and small words joined into
# a regex.
my $sw = join("|", qw(
  a about after against all among an and around as at be before between both
  but by can do down during first for from go have he her him how
  I if in into is it its last like me my new of off old
  on or out over she so such that the their there they this through to
  too under up we what when where with without you your)
);

my %sets=();    # word sets for each title.
my %titles=();  # count of how many times we see the same title.

while(<>) {
  chomp;
  # take a copy of the original input line, so we can use it as
  # a key for the hashes later.
  my $orig = $_;

  # "simplify" the input line
  s/[[:punct:]]//g;  #/ strip punctuation characters
  s/^\s*|\s*$//g;    #/ strip leading and trailing spaces
  $_=lc;             #/ lowercase everything, case is not important.
  s/\b($sw)\b//iog;  #/ optional. strip small words
  next if (/^$/);

  $sets{$orig} = Set::Tiny->new(split);
  $titles{$orig}++;
};

my @keys = (sort keys %sets);

foreach my $title (@keys) {
  next unless ($titles{$title} > 0);

  # if we have any exact dupes, print them. and make sure they won't
  # be printed again.
  if ($titles{$title} > 1) {
    print "$title\n" x $titles{$title};
    $titles{$title}  = 0;
  };

  foreach my $key (@keys) {
    next unless ($titles{$key} > 0);
    next if ($key eq $title);

    my $intersect = $sets{$key}->intersection($sets{$title});
    my $k=scalar keys %{ $intersect };

    #print STDERR "====>$k(" . join(",",sort keys %{ $intersect }) . "):$title:$key\n" if ($k > 1);

    if ($k >= 3) {
      print "$title\n" if ($titles{$title} > 0);
      print "$key\n" x $titles{$key};
      $titles{$key}   = 0;
      $titles{$title} = 0;
    };
  };
};

Save it as, e.g. blueray.pl, and make it executable with chmod +x.

Given the new sample input, it produces the following output:

$ ./blueray.pl TestData.txt 
7L: The Seven Levels of Communication
The Seven Levels of Communication: Go From Relationships to Referrals by Michael J. Maher
A History of Money and Banking in the United States: The Colonial Era to World War II
The History of Banking: The History of Banking and How the World of Finance Became What it is Today
America's Bank: The Epic Struggle to Create the Federal Reserve
America's Money Machine: The Story of the Federal Reserve
Freakonomics: A Rogue Economist
Freakonomics: A Rogue Economist Explores the Hidden
Freakonomics: A Rogue Economist Explores the Hidden
Freakonomics: A Rogue Economist Explores the Hidden Side of Everything by Steven Levitt
Money Master the Game by Tony Robbinson
Money Master the Game by Tony Robbinson
Money Master the Game by Tony Robbinson
The Federal Reserve and its Founders: Money, Politics, and Power
The Power and Independence of the Federal Reserve
Venture Deals by Brad Feld
Venture Deals by Brad Feld & Jason Mendelson
Zen and the Art of Motorcycle Maintenance: An Inquiry into Values
Zen and the Art of Motorcycle Maintenance: An Inquiry into Values

This is not exactly the same as your example output. Because it checks for the presence of common words in titles while ignoring their exact order, it is more likely to find false positives and less likely to miss matches that it shouldn't (false negatives).

If you want to experiment with this or just see what words it is matching (or almost matching) on, you can uncomment the #print STDERR line

  • I will In-Shah-Allah check the script tomorrow. If it works better then I will accept it. Can you please look at the following question as well stackoverflow.com/questions/57753634/… – blueray Sep 2 at 11:27
  • 1
    @blueray that looks like the question i started answering (but you deleted it a few minutes before i posted an early version of my answer) move over to SO. anyway, my script was written to answer that question. It also answers this one, so I posted it here. – cas Sep 2 at 11:29
  • 1
    One thing worth mentioning is that the small words list in the script above includes the word by. You can delete it from the list if you like, but the reason why it's there is to stop the script from matching on by plus any two other words - e.g. Aardvark Taxidermy for Personal Wealth by Peter Smith would match The Wealth of Nations by Adam Smith (matches on by, Wealth, and Smith). The first book is (I hope) entirely non-existent but if it did exist, it would not be at all related to an economics text. – cas Sep 2 at 11:33
  • 1
    To do "smarter" matching, you'd need to have both the title and the author for all books - either tab separated or in some structured text format like json, or in a database. – cas Sep 2 at 11:37
  • I think we made a small mistake here. The Output should only show Full Or Partial Duplicates. Full Or Partial Duplicates should be shown every time they occur. For example, if Envisioning the Survey Interview of the Future is there three times in the booklist, it should be shown three times in the output. Books that do not have any Full Or Partial Duplicates should not appear in the output. Example, Books like Blogging for Your Business, Liar’s Poker by Michael Lewis etc should not be in the output as they do not have any Full Or Partial Duplicates. – blueray Sep 2 at 14:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.