I am using rsync to back up a repository that contains many gz files including many new ones each day. The rsync backup proceeds more slowly than it should because these gz files are not built with gzip's --rsyncable option (which makes gz files much more 'rsync-friendly' without significantly increasing their size or affecting their compatibility). And I can't fix the problem at creation time because the files are generated by a python script (rdiff-backup) which uses python's gzip module and this does not support an equivalent to gzip's --rsyncable.

So before running rsync I can identify any new gz files in the source data (i.e. new since the last time rsync was run). Now I want to 're-gzip' these files so that they are gzipped in rsyncable-format. Then I can run rsync from the optimised source.

I think this means running each file through gunzip and then gzip --rsyncable but I am not too sure how to do this in a way that won't risk losing data or metadata. Suggestions gratefully received.

  • 8
    The only way --rsyncable should matter is if the files get changed between runs and rsync tries to send the changes. New files don't care if they're rsyncable or not, because rsync has to send all the data anyway. Are the files being changed between rsync runs? – Tom Hunt Nov 20 '15 at 18:39
  • Good point. Actually I'm not sure, I will check into that. Let's assume for now that yes the contents of some gz files do get changed. – gogoud Nov 20 '15 at 18:47
  • The best thing I can think of is to run a script that checks for new files, un-gzips them, then gzips them again with --rsyncable. – Tom Hunt Nov 20 '15 at 18:49
  • I agree that if the files don't change, this shouldn't be an issue. In particular, for speed make sure you skip checksumming based on time by preserving times using the -a flag. Also, my version of gzip doesn't have an --rsyncable flag, but it does come with a program called znew that could probably be used for what you need. – user3188445 Nov 20 '15 at 18:50
  • 2
    It turns out that, as Tom thought, the gz files created by rdiff-backup don't change once created and so using --rsyncable wouldn't help. I was hoping for a line of code or short script that would safely unpack a gz archive and repack it using --rsyncable. But it's just an academic question for me now. – gogoud Nov 23 '15 at 6:09
#! /bin/bash

set -euo pipefail

##  TOKEN's creation time marks the time since last recompression

if [ -f ${TOKEN} ]
    find -name '*.gz' -cnewer "${TOKEN}"
    # Process all compressed files if there is no token.
    find -name '*.gz'
fi | while read f
    # Do it in two steps
    gunzip < "$f" | gzip --rsyncable > "$f.tmp"

    # Preserve attributes
    cp "$f" "$f.tmp" --attributes-only

    # and rename atomically.
    # set -e ensures that a problem in the previous step 
    # will stop the full script. 
    mv -v "$f.tmp" "$f"

# Update the token
touch ${TOKEN}
| improve this answer | |
  • 1
    By doing gunzip | gzip, you're loosing the uncompressed name and time as stored in the gz file (and seen with gzip -vNl) – Stéphane Chazelas Sep 27 '16 at 12:04
  • @Stéphane Chazelas: You are right: if this information is relevant (it has never been relevant for me), we are losing it. Maybe the best solution would be for gunzip to directly support this re-compression. It could pass all metadata internally. – Raúl Salinas-Monteagudo Sep 28 '16 at 14:13
  • @StéphaneChazelas Do you know any to do it losslessly? – Tom Hale May 19 '18 at 5:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.