0

The Problem

I have a lot of old books that I want to scan and digitize. For this, I use some flatbed scanner, xsane and GImageReader, which works great.

Back a few years ago, when I was still using Windows for such things, I used ABBY Fine Reader, which I was also very happy with, but which is not available on Linux.

Now, comparing the PDFs I create now and the ones I created back in the day, I see that the files are much larger today.

With ABBY, I used to get PDFs with 50-60 pages with file sizes between 10-50 MB, which I find acceptable. Nowaydays, I have PDFs with 50-60 pages with file sizes of 150+ MB, which is not very usable when reading them on a smartphone for example.

I use the same settings for scanning, namely:

  • A4 pages
  • jpg compression
  • 300 dpi
  • color scans for the covers
  • greyscale scans for all interior pages

I assume that the size difference is related to ABBY using some commercial magic to be smart about image compression, which GImageReader doesn't have. Perhaps they identify non-empty areas (illustrations and text blocks) and save them at a higher quality, while aggressively compressing the "background image" or something like that - or maybe they are simply able to identify that some pages are greyscale while others are colored, a distinction which might be lost on GImageReader. I don't know, really, and I'd love to understand it.

What I have tried

Since then, I have played a bit with various methods of PDF compression. Most online guides suggest using gs or pdftk, both of which I have tried. In my specific case, I observe the following:

  • Option 1: gs for pdf->pdf. The /printer and /prepress settings do not reduce the file size at all, the /screen and /ebook settings lead to notable degradation of image quality.
  • Option 2: gs for pdf->ps and then ps->pdf. This leads to a notable reduction in filesize (I don't understand why this is any different from Option 1 but whatever) and I was happy with that option, until I noticed that apparently the glyphs of the text get lost in translation. When I copy&paste text segments from the PDF, the result is some wingdings-type gibberish, where I was able to copy&paste text from the original PDF, so this is a no-no.
  • Option 3: pdftk for pdf->pdf. This does not seem to reduce the filesize at all.

What to do now

I'm a bit lost as to how it is possible that the PDF compression techniques yield such vastly different results. I am looking for a tool that runs under linux (preferably FOSS, but I'd settle for an affordable commercial product as well) and provides significant PDF compression of scanned & OCRd PDFs without notable loss of quality against a 300dpi A4 JPG.

1

I assume that the size difference is related to ABBY using some commercial magic to be smart about image compression

Don't assume, investigate. The PDF format is well-documented, read up on the details. Open your PDF files in an editor (or just use less) and have a look at how the pages are actually encoded, and find the difference. Or install a package like mutools with command line tools that can extract parts of the PDF file.

An image in a PDF will take up a different amount of space based on the resolution it is stored with (which may or may not be the same as the one it was scanned with), and the compression algorithm.

Standard compression according to the standard are methods are

  • ASCIIHexDecode
  • ASCII85Decode
  • LZWDecode (Lempel-Ziv-Welch) .
  • FlateDecode (zlib/deflate)
  • RunLengthDecode

and a few that probably don't apply.

So find out what resolution and compression method ABBY used, then try to find tools to reproduce that (and you may need to modify existing tools if they don't do this out of the box).


From what I understand, this means that ABBY uses the JPXDecode filter with the Mask feature to encode the image, which means that I'd be looking for a linux/FOSS alternative that can do JPXDecode (JPG2000?) compression.

Exactly. Note that JPG may not be the best compression method for text, because it's geared towards photos, and that means it cannot render the sharp transitions typical for text very well. On the other hand, as these are scans, the transitions may already be inherently blurred when scanning.

Note also that JPG has quite a few parameters that influence compression ratio vs. quality.

So in that case, use mutools to extract a few of the images, use some other tool (e.g. mediainfo or identify -verbose from ImageMagick/GraphicsMagick) to find out the parameters of the JPG images.

Also have a very close look at the decompressed JPG image at high magnifications, and decide if the quality is good enough.

There should be plenty of open-source tools to create a JPG file from your scanned images in the desired resolution and quality, but I don't know any tool offhand that can pack them into a PDF.

5
  • Thanks so much for this highly insightful comment, I've started looking at the PDFs and found this. The ABBY output has 3 layers: 1 image layer with the text, 1 text layer with the text, and 1 image layer with the entirity of the page. The GImageReader output only has 2 layers, the image and the text. As for the ABBY one, here's how the stream objects look (not sure waht to make of this): – carsten Jan 3 at 15:41
  • <</Type/XObject/Subtype/Image/Interpolate true/Width 1217/Height 1742/BitsPerComponent 8 /ColorSpace/DeviceRGB/Filter[/JPXDecode]/Length 62373>>stream ^LjP <87> ^Tftypjp2 jp2 Yjp2h ^Vihdr ^F<CE> ^D<C1> ^C^G^G^A ^Ocolr^A ^P ,res ^RresdK<97><80> K<97><80> ^D^D ^RrescK<97><80> K<97><80> ^D^D jp2c<FF>O<FF>Q / ^D<C1> ^F<CE> ^D<C1> ^F<CE> ^C^G^A^A^G^A^A^G^A^A<FF>R ^L ^A^A^E^D^D <FF>\ #"w^^v<EA>v<EA>v<BC>o o n<E2>gLgLgdP^CP^CPEW<D2>W<D2>Wa<FF>d ^Q ^AKakadu-v6.3.1<FF>d Y ^AKdu-Layer-Info: log_2{Delta-D(MSE)/[2^16*Delta-L(bytes)]}, L(bytes) -54.4, 6.2e+004 <FF><90> – carsten Jan 3 at 15:42
  • <</Type/XObject/Subtype/Image/Interpolate true/Width 609/Height 871/BitsPerComponent 8/ColorSpace /DeviceRGB/Filter[/JPXDecode]/Mask 148 0 R/Length 1451>>stream ^LjP ^M \207 ^Tftypjp2 jp2 Yjp2h ^Vihdr ^Cg ^Ba ^C^G^G^A ^Ocolr^A ^P ,res ^Rresd%\314\200 %\314\200 ^D^D ^Rresc%\314\200 %\314\200 ^D^\ D jp2c\377O\377Q / ^Ba ^Cg ^Ba ^Cg ^C^G^A^A^G^A^A^G^A^A\377R ^L ^A^A^E^D^D \377\ #"w^^v\352v\352v\274o o n\342gLgLgdP^CP\ ^CPEW\322W\322Wa\377d ^Q ^AKakadu-v6.3.1\377d Y ^AKdu-Layer-Info: log_2{Delta-D(MSE)/[2^16*Delta-L(bytes)]}, L(bytes) -52.4, 1.3e+003 \377\220 – carsten Jan 3 at 15:45
  • On the other hand, the GImageReader objects look like this: obj <</Length 28 0 R/Filter /FlateDecode>> stream x\234\255\Ko$Gr\256\203O – carsten Jan 3 at 15:48
  • From what I understand, this means that ABBY uses the JPXDecode filter with the Mask feature to encode the image, which means that I'd be looking for a linux/FOSS alternative that can do JPXDecode (JPG2000?) compression. What I'm confused about is the mention of Kakadu-v6.3.1 - what is that? Is that the image compression library used by ABBY? – carsten Jan 3 at 15:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.