0

I'm trying to compare text between two versions of a PDF document. I was able to create a diff with pdftotext, format some replacements with perl and colordiff.

This output a lot of differences, but many of them are paragraphs that are identical but one paragraph has a small change in the beginning causing the rest to have different line-breaks.

Does anyone have any suggestions on how to compare the documents efficiently?

Edit: the pdfs are digitally created and contain text information, the pdfs are not scanned images.

2
  • 2
    The only idea I could think of is to assume that paragraphs are separated by at least one empty line (or some other delimiter), then convert each paragraph to a single line (by removing the newlines inside the paragraph). The problem is that for long paragraphs you'll have very long lines and it will be difficult to read the diffs. But if that's what you want, I can figure out an answer for you. I just don't want to waste time if this approach won't help with what you're trying to do. :)
    – aviro
    Jan 4 at 15:15
  • I had a similar approach. The problem with this is, that if I do it this way when most of the paragraph is the same but there's a change in the middle, the whole line will show up in the diff. I tried reflowing the lines to very short line-lengths, but the problem with this is, that if there's a change that leads to a following line being broken between different lines, theres again more lines showing up than which actually contain changes. (Maybe I need more sophisticated tools than just bash. I'll look at R and python pacakges for this)
    – snaut
    Jan 11 at 9:55

1 Answer 1

0

I'm afraid pdftotext alone is by far not sufficient to solve this.

Looking at how actual OCR software, which doesn't have the textual information ready in the document, goes at this:

  1. Create a downscaled version of the document, where "downscaling" is done by keeping the darkest pixel for every N×N piece of original image
  2. Find the largest whitespace-separated squares, to understand as paragraphs
  3. Take an educated guess on the order of these paragraphs (for example, if you have a square at the top of the page that spans > 80% of the page width, then two similarly wide blocks of roughly the same height next to each other below that, your first block is a heading, and then comes main body of text that starts with the left block and continues with the right)
  4. For each of these squares, do a similar thing with smaller blocks to find the lines of text
  5. Segment into words, and if necessary into letters, and based on an internal model of what is probable to appear in sentences, and on similarity of these letters and words to stored reference letters, convert these pixels to text.

Obviously, you're done after the 3. step. And it's not too hard to implement yourself!

So, you start by converting your image into a somewhat high-resolution image (300dpi), e.g. using `mutool draw -r 300 -c gray -o 'rastered_%03d.png' (basically, step 0 above, get a scan of your document).

Then, for each resulting rastered_${pagenumber}.png, you do the downscaling (step 1) and paragraph segmentation (step 2). Then sort probably simply by top left corner (y first, then x) (step 3).

After the third step, you just use the coordinates of the squares with pdftotext -x ${top_left_x} -y ${top_left_y} -W ${width} -H ${height}. And then concatenate the resulting text in the right order :)

2
  • Sorry, my question was not clear. My pdfs contain text not images, they are digitally created and text extraction is not the problem, just that the line-breaks are hardcoded as it often is with pdfs.
    – snaut
    Jan 11 at 9:57
  • yes, that's exactly what I address in my answer. Remove all line breaks that are not inherent to the paragraph layout; for that you need to learn the paragraph layout first. Jan 11 at 12:11

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .