I am using rsync to back up a repository that contains many gz files including many new ones each day. The rsync backup proceeds more slowly than it should because these gz files are not built with gzip's --rsyncable option (which makes gz files much more 'rsync-friendly' without significantly increasing their size or affecting their compatibility). And I can't fix the problem at creation time because the files are generated by a python script (rdiff-backup) which uses python's gzip module and this does not support an equivalent to gzip's --rsyncable.

So before running rsync I can identify any new gz files in the source data (i.e. new since the last time rsync was run). Now I want to 're-gzip' these files so that they are gzipped in rsyncable-format. Then I can run rsync from the optimised source.

I think this means running each file through gunzip and then gzip --rsyncable but I am not too sure how to do this in a way that won't risk losing data or metadata. Suggestions gratefully received.

  • 8
    The only way --rsyncable should matter is if the files get changed between runs and rsync tries to send the changes. New files don't care if they're rsyncable or not, because rsync has to send all the data anyway. Are the files being changed between rsync runs?
    – Tom Hunt
    Nov 20, 2015 at 18:39
  • Good point. Actually I'm not sure, I will check into that. Let's assume for now that yes the contents of some gz files do get changed.
    – gogoud
    Nov 20, 2015 at 18:47
  • The best thing I can think of is to run a script that checks for new files, un-gzips them, then gzips them again with --rsyncable.
    – Tom Hunt
    Nov 20, 2015 at 18:49
  • I agree that if the files don't change, this shouldn't be an issue. In particular, for speed make sure you skip checksumming based on time by preserving times using the -a flag. Also, my version of gzip doesn't have an --rsyncable flag, but it does come with a program called znew that could probably be used for what you need. Nov 20, 2015 at 18:50
  • 2
    It turns out that, as Tom thought, the gz files created by rdiff-backup don't change once created and so using --rsyncable wouldn't help. I was hoping for a line of code or short script that would safely unpack a gz archive and repack it using --rsyncable. But it's just an academic question for me now.
    – gogoud
    Nov 23, 2015 at 6:09

1 Answer 1

#! /bin/bash

set -euo pipefail

##  TOKEN's creation time marks the time since last recompression

if [ -f ${TOKEN} ]
    find -name '*.gz' -cnewer "${TOKEN}"
    # Process all compressed files if there is no token.
    find -name '*.gz'
fi | while read f
    # Do it in two steps
    gunzip < "$f" | gzip --rsyncable > "$f.tmp"

    # Preserve attributes
    cp "$f" "$f.tmp" --attributes-only

    # and rename atomically.
    # set -e ensures that a problem in the previous step 
    # will stop the full script. 
    mv -v "$f.tmp" "$f"

# Update the token
touch ${TOKEN}
  • 1
    By doing gunzip | gzip, you're loosing the uncompressed name and time as stored in the gz file (and seen with gzip -vNl) Sep 27, 2016 at 12:04
  • @Stéphane Chazelas: You are right: if this information is relevant (it has never been relevant for me), we are losing it. Maybe the best solution would be for gunzip to directly support this re-compression. It could pass all metadata internally. Sep 28, 2016 at 14:13
  • @StéphaneChazelas Do you know any to do it losslessly?
    – Tom Hale
    May 19, 2018 at 5:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .