4

I have a pdf document with over 200 duplicate pages among the total 900 of the document. When there is a duplicate, it appears immediately after the original.

Maybe with pdftk the job can be done, but I need some way to find out the duplicates...

3 Answers 3

5

comparepdf is a command line tool for comparing PDFs. The exit code is 0 if the files are identical and non-zero otherwise. You may compare by text content or visually (interesting for e.g. scans):

comparepdf 1.pdf 2.pdf
comparepdf -ca 1.pdf 2.pdf #compare appearance instead of text

So what you could do is explode the PDF, then compare pairwise and delete accordingly:

#!/bin/bash
#explode pdf
pdftk original.pdf burst
#compare 900 pages pairwise
for (( i=1 ; i<=899 ; i++ )) ; do
  #pdftk's naming is pg_0001.pdf, pg_0002.pdf etc.
  pdf1=pg_$(printf 04d $i).pdf
  pdf2=pg_$(printf 04d $((i+1))).pdf
  #Remove first file if match. Loop not forwarded in case of three or more consecutive identical pages 
  if comparepdf $pdf1 $pdf2 ; then
     rm $pdf1
  fi
done
#renunite in sorted manner:
pdftk $(find -name 'pg_*.pdf' | sort ) cat output new.pdf

EDIT: Following @notautogenerated's remark, one might be bettor off selecting pages from the orginal file instead of unifying single-page PDFs. After the pairwise comparison is done, one could do the following:

pdftk original.pdf cat $(find -name 'pg_*.pdf' |
                        awk -F '[._]' '{printf "%d\n",$3}' |
                        sort -n ) output new.pdf
5
  • 1
    @AdminBee Added reunite - good point. While I agree it is good practice to quote the variables, the strict naming pattern of pdftk + burst does render it unnecessary here (and also helps in the sorted renuniting, i.e. no -print0 in find and all). .... but well, "no quotes for this variable needed" are some the most famous last words, aren't they?
    – FelixJN
    Jun 21, 2021 at 14:04
  • That was indeed my concern, well stated ;)
    – AdminBee
    Jun 21, 2021 at 14:06
  • It would be slightly better to record which pages should be added and then use that as a parameter to cat (e.g. pdftk original.pdf cat $pages output new.pdf). This would avoid making multiple copies of fonts and other global resources that pdftk is not smart enough to deduplicate, resulting in a smaller output size. Jun 21, 2021 at 17:33
  • @notautogenerated good point. Added.
    – FelixJN
    Jun 21, 2021 at 21:21
  • Thanks! it gave me the idea, but I couldn't install the comparepdf tool. I managed to do the job in other way. You forgot the k of pdftk in your last command :)
    – fich
    Jun 22, 2021 at 11:21
1

The following is a modified version of @FelixJN's code, where I fixed typos in the format string of printf. The code was verfied by me and it was working.

#!/bin/bash
pdftk original.pdf burst  #explode the pdf
#the resulting files are named as  pg_0001.pdf, pg_0002.pdf etc.

for (( i=1 ; i<=1140 ; i++ )) ; do #loop over all the signle-page pdf files
  pdf1=pg_$(printf %04d $i).pdf
  pdf2=pg_$(printf %04d $((i+1))).pdf
  echo $pdf1 $pdf2
  if comparepdf $pdf1 $pdf2 ; then
     rm $pdf1  #remove the first if two adjacent files are duplicate
  fi
done
#merge the remained files in sorted manner:
pdftk $(find -name 'pg_*.pdf' | sort ) cat output new.pdf

0

If you are unable to use the comparepdf tool, the following was a valid solution for me (using FelixJN's answer):

#explode pdf
pdftk original.pdf burst

#delete consecutive pages that have the same size        
last=-1; find . -type f -name '*.pdf' -printf '%f\0' | sort -nz | 
    while read -d '' i; do 
        s=$(stat -c '%s' "$i"); 
        [[ $s = $last ]] && rm "$i"; 
    last=$s; 
done

#rearrange the pdf
pdftk original.pdf cat $(find -name 'pg_*.pdf' |
                        awk -F '[._]' '{printf "%d\n",$3}' |
                        sort -n ) output new.pdf

this may delete a page that is not meant to be deleted, but the probability i think is low. Source for deleting files of the same size: How to remove the same size files in a directory?

1
  • 1
    Maybe instead of using the size only, you could use cmp instead. It compares files byte-wise. Then limit yourself to the e.g. first 100k bytes: cmp -n 100k pg_0001.pdf pg_0002.pdf (Exit code 0 for identical, 1 for mismatch). Probably the identical pages will differ in the ID-field only, see diff -a pg_0001.pdf pg_0002.pdf (for identical pages). E.g. this might also work: cmp <(grep -av 'ID' pg_0001.pdf) <(grep -av 'ID' pg_0002.pdf). But this depends a bit on the PDF version and how duplicate pages actually came into existence.
    – FelixJN
    Jun 23, 2021 at 13:39

This site is temporarily in read-only mode and not accepting new answers.

Not the answer you're looking for? Browse other questions tagged .