I would like to create a compressed mirror of some directories in such a way that it can be rsynced again, both from the original destination and to a remote destination.
I can do it inefficiently in two steps:
rsync -a /source/ /compressed-mirror/
gzip --recursive --rsyncable --force /compressed-mirror/
# later: rsync -a /compressed-mirror/ remote:/another-compressed-mirror/
The problem is that if I rerun the first rsync to update the mirror, it needs to copy and gzip all of the source files all over again, even if only a few of them changed.
Question: Is there some other way of doing this that would minimize the amount of data transfer and hard drive use needed on every update?
Notes:
I am not stuck on gzip for compression - I only chose it because it allows the compressed files to be rsynced.
The purpose of the local compressed mirror is to minimize the "network up" time needed to do the rsync to the remote mirror, as both the internet connection and power state of the source machine (a laptop) is unreliable. Some parts of the local mirror get encrypted before rsync to the remote mirror. The remote mirror then gets versioned with rsnapshot and uploaded to another remote server over a much more stable internet connection.
Updates/ ideas:
- Use filesystem compression (see comments by cas) - but this forces rsync to re-compress the files when transferring them to the remote server, which seems redundant.
- Keep the rsync'd directory uncompressed. Each time rsync updates it, record the files that changed. For each changed file, individually create a compressed copy (or delete it) in the compressed mirror. But the next idea does the same thing more efficiently:
- Keep a list of file paths and file sizes (or checksums) in the source. At each update, make a new list and compare it with the old list. For each changed file, individually create a gzipped copy (or delete it) in the first compressed mirror by using
gzip < source/path/file > mirror/path/file.gz
. This seems to be the most efficient solution so far.
rsync
from upstream, and re-compress them. You'd also need to tellrsync
to use checksums to skip files (rsync -c
), both with upstream and downstream mirrors. – Satō Katsura Jul 21 '16 at 3:28e2compr
patch forext2
(and ane3compr
patch forext3
), but they haven't been updated for years, probably because the existence of btrfs makes them obsolete....if you want advanced features like that in a fs, use an advanced fs. NOTE: you can use a compressed fs for the target fs OR both the source and target filesystems. There's little point in doing it for only the source fs. – cas Jul 21 '16 at 8:41lz4
- an extremely fast compression and decompression algorithm that favours speed over compression ratios. – cas Jul 21 '16 at 13:54compression=on | off | lzjb | lz4 | gzip | gzip-N | zle
- pthree.org/2012/12/18/… – cas Jul 21 '16 at 13:56