How to convert existing gz (gzip) files to rsyncable

Question

I am using rsync to back up a repository that contains many gz files including many new ones each day. The rsync backup proceeds more slowly than it should because these gz files are not built with gzip's --rsyncable option (which makes gz files much more 'rsync-friendly' without significantly increasing their size or affecting their compatibility). And I can't fix the problem at creation time because the files are generated by a python script (rdiff-backup) which uses python's gzip module and this does not support an equivalent to gzip's --rsyncable.

So before running rsync I can identify any new gz files in the source data (i.e. new since the last time rsync was run). Now I want to 're-gzip' these files so that they are gzipped in rsyncable-format. Then I can run rsync from the optimised source.

I think this means running each file through gunzip and then gzip --rsyncable but I am not too sure how to do this in a way that won't risk losing data or metadata. Suggestions gratefully received.

Answer 1

#! /bin/bash

set -euo pipefail

##  TOKEN's creation time marks the time since last recompression
TOKEN=.lastRecompression   

if [ -f ${TOKEN} ]
then
    find -name '*.gz' -cnewer "${TOKEN}"
else
    # Process all compressed files if there is no token.
    find -name '*.gz'
fi | while read f
do
    # Do it in two steps
    gunzip < "$f" | gzip --rsyncable > "$f.tmp"

    # Preserve attributes
    cp "$f" "$f.tmp" --attributes-only

    # and rename atomically.
    # set -e ensures that a problem in the previous step 
    # will stop the full script. 
    mv -v "$f.tmp" "$f"
done

# Update the token
touch ${TOKEN}