Proper way to create a compressed, rsyncable mirror of a directory?

Question

I would like to create a compressed mirror of some directories in such a way that it can be rsynced again, both from the original destination and to a remote destination.

I can do it inefficiently in two steps:

rsync -a /source/ /compressed-mirror/
gzip --recursive --rsyncable --force /compressed-mirror/
# later: rsync -a /compressed-mirror/ remote:/another-compressed-mirror/

The problem is that if I rerun the first rsync to update the mirror, it needs to copy and gzip all of the source files all over again, even if only a few of them changed.

Question: Is there some other way of doing this that would minimize the amount of data transfer and hard drive use needed on every update?

Notes:

• I am not stuck on gzip for compression - I only chose it because it allows the compressed files to be rsynced.

• The purpose of the local compressed mirror is to minimize the "network up" time needed to do the rsync to the remote mirror, as both the internet connection and power state of the source machine (a laptop) is unreliable. Some parts of the local mirror get encrypted before rsync to the remote mirror. The remote mirror then gets versioned with rsnapshot and uploaded to another remote server over a much more stable internet connection.

Updates/ ideas:

• Use filesystem compression (see comments by cas) - but this forces rsync to re-compress the files when transferring them to the remote server, which seems redundant.
• Keep the rsync'd directory uncompressed. Each time rsync updates it, record the files that changed. For each changed file, individually create a compressed copy (or delete it) in the compressed mirror. But the next idea does the same thing more efficiently:
• Keep a list of file paths and file sizes (or checksums) in the source. At each update, make a new list and compare it with the old list. For each changed file, individually create a gzipped copy (or delete it) in the first compressed mirror by using gzip < source/path/file > mirror/path/file.gz. This seems to be the most efficient solution so far.

Answer 1

The most efficient way of handling your requirement may be NOT to compress the intermediate mirror target. This will allow rsync to use its delta algorithm between the local and remote host to transfer only changed data. (The delta algorithm does not apply when copying between two directories on the same host, though.)

# Any changed files will be copied completely, even if only one byte changed
rsync -a --delete /source/ /mirror/

# Only copy changed parts of changed files
rsync -az --delete /mirror/ remote:/mirror/

The most efficient solution (in terms of hit on the local hard disk and intermediate network) would be to dispense with the local mirror completely, and just copy directly from your source to the remote destination:

# Only copy changed parts of changed files
rsync -az --delete /source/ remote:/mirror/