Find duplicate PDF files by content

Question

Some journals generate a different PDF for each download. APS for example stores time and the IP address in the PDF.

Or there is a paper version with hyper links and one with text references.

How is it possible to find duplicate downloads of papers with 90 % equal content on a linux system by using open source software?

I have been thinking about converting the PDF files to plain text in a temporary directory with pdf2txt. Then I could filter all filenames which diff a b results more than x lines. But this is not elegant at all and will fail with scanned publications. Journals often do not provide OCR text for old publications.

I also tried compare in the ImageMagick suite, but I could not handle multipage PDF files with this tool.

diffpdf 2.1.1 does a good job in a GUI on two files, but I could not figure out how to apply it on many files, and recent versions are not available under any open source license.

Answer 1

Since different publishers use different methods of "marking" the PDFs you need to make sure you compare without taking the markings into account.

You also need an efficient method to compare a new PDF to all already downloaded PDFs in case you repeatedly download the same PDF and it is e.g. marked with the IP and/or date-time-stamp as you suggest. You don't want to use a time consuming comparison mechanism that compares each new PDF with many already downloaded PDFs

What you need is a utility that strips each of the possible markings and generate a hash of the remaining data. You will need to keep a hash → file name map, which can be in a simple file, and if a computed hash is already in the file you have a duplicate (and delete it or do whatever needed) and if the hash in not yet there, you add the hash and file name. The file would look something like:

6fcb6969835d2db7742e81267437c432  /home/anthon/Downloads/explanation.pdf
fa24fed8ca824976673a51803934d6b9  /home/anthon/orders/your_order_20150320.pdf

That file is negligently small compared to the original PDFs. If you have millions of PDFs you might consider storing this data in a database. For efficiency sake you might want to include the filesize and number of pages in there (pdfinfo | egrep -E '^Pages:' | grep -Eo '[0-9]*').

---

The above pushes the problem to removing the markings and generating the hash. If you know where the PDF comes from when invoking the hash generating routine (i.e. if you do the downloads programmatically), you can fine-tune the hash generation based on that. But even without that there are several possibilities for hash generation:

1. if the metadata for title and author is non-empty and does not including non-specific strings like "Acrobat" or "PDF" you could generate the hash based on just the author and title information. Use pdfinfo -E file.pdf | grep -E '^(Author:)|(Title:) | md5sum to get the hash. You can include the number of pages in calculating the hash as well ('Pages:' in the pdfinfo output).
2. if the previous rule doesn't work and the PDF contains images, extract the images and generate a hash on the combined image data. If the images ever contain text in the footer or header like "Licensed to Joe User", strip an X number of lines form the top or bottom, before calculating the hash. If that markings is in some big lettered grayed background text this will of course not work, unless you filter out pixels that are not totally black (for that you could use imagemagick). You can use pdfimages to extract the image information into a temporary file.
3. if the previous rules don't work (because there are no images) you can use pdftext to extract the text, filter out the marking (if you filter out a little to much, that is not a problem) and then generate the hash based on that.

Additionally you can compare if the file size of the old file found via the hash and see if is within certain margins with the new file. Compression and ifferences in strings (IP/date-time-stamp) should only result in less than one percent difference.

If you know the method the publisher uses when determining the hash, you can directly apply the "right" method of the above, but even without that you can check for the metadata and apply some heuristics, or determine the number of images in a file and compare that with the number of pages (if they are close you probably have a document consisting of scans). pdftext on scanned image PDFs also has a recognisable output.

---

As a basis to work from I created a python package that is on bitbucket and/or can be installed from PyPI using pip install ruamel.pdfdouble. This provides you with the pdfdbl command that does the scanning as described above on metadata, extracted images or on text. It doesn't do any filtering of markings (yet), but the readme describes which (two) methods to enhance to do add that.

The included readme:

ruamel.pdfdouble

this package provides the pdfdbl command:

pdfdbl scan dir1 dir2

This will walk down the directories provided as argument and for the PDF files found, create a hash based on (in order):

• metadata if unique
• images if the number of images
• text

This assumes that pdfinfo, pdfimages and pdftotext` from the poppler-utils package are avaialable.

A "database" is build up in ~/.config/pdfdbl/pdf.lst against which further scans are tested.

Removing markings

In ruamel/pdfdouble/pdfdouble.py there are two methods that can be enhanced to filter out markings in the PDF that make them less unique and make virtually the same files to have different hashes.

For text the method PdfData.filter_for_marking should be extended to remove and markings from the string that is its arguments and return the result.

For scanned images the method PdfData.process_image_and_update needs to be enhanced, e.g. by cutting off the images bottom and top X lines, and by removing any gray background text by setting all black pixels to white. This function needs to update the hash passed in using the .update() method passing in the filtered data.

Restrictions

The current "database" cannot handle paths that contain newlines

This utility is currently Python 2.7 only.

---

IP conforming stringparts can be substituted with Python's re module:

import re
IPre = re.compile("(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}"
              "([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])")

x = IPre.sub(' ', 'abcd 132.234.0.2 ghi')
assert x == 'abcd   ghi'

Answer 2

I'd give pdftotext another chance, at least for the PDFs in your collection that actually have text (otherwise you'd need to run OCR), using a better tool to process the output.

Once you have your (dirty) text output, run it through a program designed to determine similarities (rather than diff's line-by-line differences, which would be a quick path to insanity).

Consider something like the perl's String::Similarity or the simhash program (which is available in Debian but not Fedora/RHEL).

Answer 3

The PDFs contain metadata and I just checked a number of physics related papers from different publishers and they all have at least the "Title" attribute. For some, the title is the actual title of the publication, for some it contains the DOI or similar identifieres. Anyway, every paper I checked contains the title, and it is always something unique to the given publication.

You could use pdftk to access the metadata of the PDFs and compare those. For your purpose, this should definately be sufficient and is a lot quicker than pdftotext if performance is an issue. In case a paper really should not have title metadata you could still fall back to to pdftotext.

To dump all metadata to a textfile (or stdout) for further processing use

pdftk <PDF> dump_data output <TEXTFILE>

or refer to the manual for further options.

If you would like to try ImageMagick's compare but multiple pages cause a problem, you could also use pdftk to extract single pages and compare all of them separately (maybe just comparing a single one is enough, though).

Here is a code snippet that uses this approach to create a diff-like PDF output for multipage PDFs: https://gist.github.com/mpg/3894692

Answer 4

Have you looked into PDF Content Comparer? There are command line options which should let you automate the process.

You could run some sort of logic on the difference log it creates to see how similar they are.

Failing that you might try splitting the PDF's into multiple files temporarily and comparing them that way. You'd probably still have duplicates that way, though. One PDF may just have an extra blank page or something which would cause all subsequent pages to compare as being completely different.

Answer 5

Following a humble contribution to the discussion (partial answer):

After converted to text I would use the following to calculate the (word difference based) file smilarity:

wdiff -s -123 file1.txt file2.txt |    ## word difference statistics (1)
     grep -Po '(\d+)(?=% common)' |    ## 
     awk '{a+=$1}END{print a/2}'       ## (2)

(1) produces a result like

file1.txt: 36 words  33 92% common  3 8% deleted  0 0% changed
file2.txt: 35 words  33 94% common  2 6% inserted  0 0% changed

(2) = 93

Answer 6

I have a script which looks at a pdf and first tries to extract text using pdftotext, but if this fails (as it will with a scanned document), it uses ghostscript to turn a multi-page scanned pdf into a series of png files and then uses tesseract to convert this series into a single text file. If the scan is of sufficient quality it does a pretty good job. It would be straightforward to add code comparing the text between files but I haven't had this requirement.

ghostscript and tesseract are both open source and work from the command line.

Answer 7

I would offer perl as a solution. There's a module called CAM::PDF which allows you to extract ... PDF content.

It works a little like this:

#!/usr/bin/perl

use strict;
use warnings;

use CAM::PDF;

my $file = 'sample.pdf';

my $pdf = CAM::PDF->new($file);

my $word_count = 0;
for my $pagenum ( 1 .. $pdf->numPages ) {
    my $page_text = $pdf->getPageText($pagenum) );
    print $page_text; 
}

You can extract the text and compare that.

For scanned only documents - it's much harder, but assuming they are using the same base images (e.g. haven't separately scanned them) then you can probably use:

#!/usr/bin/perl

use strict;
use warnings;

use CAM::PDF;
use CAM::PDF::Renderer::Images;
use Data::Dumper; 

my $file = 'sample.pdf';

my $pdf = CAM::PDF->new($file);

my $word_count = 0;
for my $pagenum ( 1 .. $pdf->numPages ) {
    my $content =  $pdf->getPageText($pagenum);
    my $page = $pdf->getPageContentTree($pagenum);
    my $gs = $page->findImages();
    my @imageNodes = @{$gs->{images}};
    print Dumper \@imageNodes;

    print Dumper \$gs;
}

I've not tested it particularly well, because I don't have your source documents. I think this approach should do the trick though - you're not comparing actual image content, because .... well, that's really difficult. But you should be able to recognise similar images from the metadata.

For identical PDFs with different metadata, then something simple like hashing the text content and the image metadata should do the trick.

Answer 8

There is an Linux application, called recoll. It can perform the task, but only for pdfs with text layer.