3

I would like to create a compressed mirror of some directories in such a way that it can be rsynced again, both from the original destination and to a remote destination.

I can do it inefficiently in two steps:

rsync -a /source/ /compressed-mirror/
gzip --recursive --rsyncable --force /compressed-mirror/
# later: rsync -a /compressed-mirror/ remote:/another-compressed-mirror/

The problem is that if I rerun the first rsync to update the mirror, it needs to copy and gzip all of the source files all over again, even if only a few of them changed.

Question: Is there some other way of doing this that would minimize the amount of data transfer and hard drive use needed on every update?

Notes:

  • I am not stuck on gzip for compression - I only chose it because it allows the compressed files to be rsynced.

  • The purpose of the local compressed mirror is to minimize the "network up" time needed to do the rsync to the remote mirror, as both the internet connection and power state of the source machine (a laptop) is unreliable. Some parts of the local mirror get encrypted before rsync to the remote mirror. The remote mirror then gets versioned with rsnapshot and uploaded to another remote server over a much more stable internet connection.

Updates/ ideas:

  • Use filesystem compression (see comments by cas) - but this forces rsync to re-compress the files when transferring them to the remote server, which seems redundant.
  • Keep the rsync'd directory uncompressed. Each time rsync updates it, record the files that changed. For each changed file, individually create a compressed copy (or delete it) in the compressed mirror. But the next idea does the same thing more efficiently:
  • Keep a list of file paths and file sizes (or checksums) in the source. At each update, make a new list and compare it with the old list. For each changed file, individually create a gzipped copy (or delete it) in the first compressed mirror by using gzip < source/path/file > mirror/path/file.gz. This seems to be the most efficient solution so far.
Improve this question
8
  • You can't do both at the same time. One way that minimises transfer traffic but not the disk space is to uncompress the files on the mirror, rsync from upstream, and re-compress them. You'd also need to tell rsync to use checksums to skip files (rsync -c), both with upstream and downstream mirrors.
    – Satō Katsura
    Jul 21 '16 at 3:28
  • the easiest way to do this is to use a filesystem that supports transparent compression - e.g. btrfs or zfs. There used to be an e2compr patch for ext2 (and an e3compr patch for ext3), but they haven't been updated for years, probably because the existence of btrfs makes them obsolete....if you want advanced features like that in a fs, use an advanced fs. NOTE: you can use a compressed fs for the target fs OR both the source and target filesystems. There's little point in doing it for only the source fs.
    – cas
    Jul 21 '16 at 8:41
  • @cas A great alternative! But when I run the second rsync to update a remote mirror, in order to copy each changed file, won't the fs have to decompress the file, rsync have to compress and decompress (-z) it, then the remote fs have to compress the file again? With gzip I was hoping to avoid redundant compression.
    – Oleg
    Jul 21 '16 at 13:48
  • uncompressing a file is often faster than reading the uncompressed version of the file (and the bigger the file, the more pronounced that is - disk i/o is slower than decompression). I can't remember what compression btrfs uses, but zfs recommends using lz4 - an extremely fast compression and decompression algorithm that favours speed over compression ratios.
    – cas
    Jul 21 '16 at 13:54
  • ah, btrfs also offers zlib (slower, better compression) and lzo (faster, worse compression). btrfs.wiki.kernel.org/index.php/Compression. zfs offers compression=on | off | lzjb | lz4 | gzip | gzip-N | zle - pthree.org/2012/12/18/…
    – cas
    Jul 21 '16 at 13:56
2

The most efficient way of handling your requirement may be NOT to compress the intermediate mirror target. This will allow rsync to use its delta algorithm between the local and remote host to transfer only changed data. (The delta algorithm does not apply when copying between two directories on the same host, though.)

# Any changed files will be copied completely, even if only one byte changed
rsync -a --delete /source/ /mirror/

# Only copy changed parts of changed files
rsync -az --delete /mirror/ remote:/mirror/

The most efficient solution (in terms of hit on the local hard disk and intermediate network) would be to dispense with the local mirror completely, and just copy directly from your source to the remote destination:

# Only copy changed parts of changed files
rsync -az --delete /source/ remote:/mirror/
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.