I have a pdf document with over 200 duplicate pages among the total 900 of the document. When there is a duplicate, it appears immediately after the original.

Maybe with pdftk the job can be done, but I need some way to find out the duplicates...


comparepdf is a command line tool for comparing PDFs. The exit code is 0 if the files are identical and non-zero otherwise. You may compare by text content or visually (interesting for e.g. scans):

comparepdf 1.pdf 2.pdf
comparepdf -ca 1.pdf 2.pdf #compare appearance instead of text

So what you could do is explode the PDF, then compare pairwise and delete accordingly:

#explode pdf
pdftk original.pdf burst
#compare 900 pages pairwise
for (( i=1 ; i<=899 ; i++ )) ; do
  #pdftk's naming is pg_0001.pdf, pg_0002.pdf etc.
  pdf1=pg_$(printf 04d $i).pdf
  pdf2=pg_$(printf 04d $((i+1))).pdf
  #Remove first file if match. Loop not forwarded in case of three or more consecutive identical pages 
  if comparepdf $pdf1 $pdf2 ; then
     rm $pdf1
#renunite in sorted manner:
pdftk $(find -name 'pg_*.pdf' | sort ) cat output new.pdf

EDIT: Following @notautogenerated's remark, one might be bettor off selecting pages from the orginal file instead of unifying single-page PDFs. After the pairwise comparison is done, one could do the following:

pdftk original.pdf cat $(find -name 'pg_*.pdf' |
                        awk -F '[._]' '{printf "%d\n",$3}' |
                        sort -n ) output new.pdf
  • 1
    @AdminBee Added reunite - good point. While I agree it is good practice to quote the variables, the strict naming pattern of pdftk + burst does render it unnecessary here (and also helps in the sorted renuniting, i.e. no -print0 in find and all). .... but well, "no quotes for this variable needed" are some the most famous last words, aren't they?
    – FelixJN
    Jun 21 '21 at 14:04
  • That was indeed my concern, well stated ;)
    – AdminBee
    Jun 21 '21 at 14:06
  • It would be slightly better to record which pages should be added and then use that as a parameter to cat (e.g. pdftk original.pdf cat $pages output new.pdf). This would avoid making multiple copies of fonts and other global resources that pdftk is not smart enough to deduplicate, resulting in a smaller output size. Jun 21 '21 at 17:33
  • @notautogenerated good point. Added.
    – FelixJN
    Jun 21 '21 at 21:21
  • Thanks! it gave me the idea, but I couldn't install the comparepdf tool. I managed to do the job in other way. You forgot the k of pdftk in your last command :)
    – fich
    Jun 22 '21 at 11:21

If you are unable to use the comparepdf tool, the following was a valid solution for me (using FelixJN's answer):

#explode pdf
pdftk original.pdf burst

#delete consecutive pages that have the same size        
last=-1; find . -type f -name '*.pdf' -printf '%f\0' | sort -nz | 
    while read -d '' i; do 
        s=$(stat -c '%s' "$i"); 
        [[ $s = $last ]] && rm "$i"; 

#rearrange the pdf
pdftk original.pdf cat $(find -name 'pg_*.pdf' |
                        awk -F '[._]' '{printf "%d\n",$3}' |
                        sort -n ) output new.pdf

this may delete a page that is not meant to be deleted, but the probability i think is low. Source for deleting files of the same size: How to remove the same size files in a directory?

  • 1
    Maybe instead of using the size only, you could use cmp instead. It compares files byte-wise. Then limit yourself to the e.g. first 100k bytes: cmp -n 100k pg_0001.pdf pg_0002.pdf (Exit code 0 for identical, 1 for mismatch). Probably the identical pages will differ in the ID-field only, see diff -a pg_0001.pdf pg_0002.pdf (for identical pages). E.g. this might also work: cmp <(grep -av 'ID' pg_0001.pdf) <(grep -av 'ID' pg_0002.pdf). But this depends a bit on the PDF version and how duplicate pages actually came into existence.
    – FelixJN
    Jun 23 '21 at 13:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.