I would like to create a compressed mirror of some directories in such a way that it can be rsynced again, both from the original destination and to a remote destination.
I can do it inefficiently in two steps:
rsync -a /source/ /compressed-mirror/
gzip --recursive --rsyncable --force /compressed-mirror/
# later: rsync -a /compressed-mirror/ remote:/another-compressed-mirror/
The problem is that if I rerun the first rsync to update the mirror, it needs to copy and gzip all of the source files all over again, even if only a few of them changed.
Question: Is there some other way of doing this that would minimize the amount of data transfer and hard drive use needed on every update?
I am not stuck on gzip for compression - I only chose it because it allows the compressed files to be rsynced.
The purpose of the local compressed mirror is to minimize the "network up" time needed to do the rsync to the remote mirror, as both the internet connection and power state of the source machine (a laptop) is unreliable. Some parts of the local mirror get encrypted before rsync to the remote mirror. The remote mirror then gets versioned with rsnapshot and uploaded to another remote server over a much more stable internet connection.
- Use filesystem compression (see comments by cas) - but this forces rsync to re-compress the files when transferring them to the remote server, which seems redundant.
- Keep the rsync'd directory uncompressed. Each time rsync updates it, record the files that changed. For each changed file, individually create a compressed copy (or delete it) in the compressed mirror. But the next idea does the same thing more efficiently:
- Keep a list of file paths and file sizes (or checksums) in the source. At each update, make a new list and compare it with the old list. For each changed file, individually create a gzipped copy (or delete it) in the first compressed mirror by using
gzip < source/path/file > mirror/path/file.gz. This seems to be the most efficient solution so far.