2

A service on a linux server is only able to do full backups, where each backup is a .tar archive (no compression). Many contents of the archive do not change from day to day. Each .tar file size is about 3GB (slowly increasing from day to day).

I want to transfer the backups to another server, which archives them. The transfer is done through the internet.

A requirement is that the backups are not altered (the result is again a list of .tar files, whose md5 sum is still identical to the original files on the server).

I'm currently using rsync to transfer the files, which works great, but all files are transferred with their full size. As far as I know rsync does some kind of deduplication on transfers, but only on a per-file level (right?).

Is there any way to transfer few similar files through a SSH connection without retransmitting identical chunks of the files (so some kind of reduplication), that

  • does not require write access on the server (no unpacking of the tar files)
  • is tolerant to connection losses (does not leave temp files on abortions and detects not correctly transmitted files)
  • is able to resume the transfer after connection losses (do not retransmit all files if connection aborts)
  • does not require any additional tools on the server (besides the standard unix toolchain including rsync)
  • still uses a client-initiated SSH connection for the transfer
  • 1
    Have you tried the -z-option of rsync? – Jodka Lemon Jul 23 '15 at 10:05
  • @JodkaLemon yes, it's still transferring all files – muffel Jul 23 '15 at 10:08
  • @Jodka - did you mean the -y option? "-y, --fuzzy find similar file for basis if no dest file" - not exactly sure what counts as "a similarly-named file" for this purpose, though. – Toby Speight Jul 23 '15 at 11:46
  • Link - what is a 'similar named' file? - implies that it looks for the closest by Levenshtein Distance, up to a limit of 25. – Toby Speight Jul 23 '15 at 12:20
2

One thing you might do is to (on the receiving side) copy the last backup file to the new name before starting rsync. Then it will transfer only the diffs between what you have and what you should have.

If you do this, be careful if you have rsync -u (update only, based on timestamp) that you ensure that your copy is older than the new source file.

  • good idea, but how should I automate that? – muffel Jul 23 '15 at 10:28
  • I was assuming that the file name would be predictable (e.g. a known prefix plus a date stamp), so you would be able to do something like cp -n $(date +'backup-%F') $(date -d yesterday +'backup-%F') immediately before your rsync. – Toby Speight Jul 23 '15 at 11:51
  • the file name correspond to the unix file stamp when the backup was created. unfortunately that differs as it depends on the system load. – muffel Jul 23 '15 at 11:52
  • In that case, you'll probably want to find the newest (or last-by-name) file on each side, and use that to decide whether and what to copy. – Toby Speight Jul 23 '15 at 12:18
2

A possible idea, from Roland (Cybso) Tapken, is to use rsync --fuzzy with per-date directories

Put the archive in directories named by the date and called current.tar and parent.tar, where parent.tar is a hardlink to the previous current.tar. In conjunction with --hard-links this should work, since the Levenshtein Distance of current.tar and parent.tar is below 25 (actually it is 3). Of course, source and target filesystems have to support hardlinks, and you must ensure that parent.tar is transmitted before current.tar.

I interpret this as (completely untested):

cd $backups
dir=$(date +%s)
latest=$(ls | tail -n 1)   # N.B. we control names in this directory!
mkdir $dir
ln $latest/01-current.tar $dir/01-parent.tar
$do_backup > $dir/02-current.tar
rsync -arz --fuzzy --hard-links ./ $server/backups/

The rsync line above can be re-executed if it fails, and it should pick up where it left off. I named the files in the daily directory with numeric prefixes to encourage the transfer of parent file first. I purposefully didn't include --delete, so that fully transferred directories could safely be removed from the client and the backups would remain on the server.

You should end up with a directory structure like this:

$backups
  1437502724
    02-current.tar-\
  1437589112       |
    01-parent.tar--/
    02-current.tar---\
  1437675488         |
    01-parent.tar----/
    02-current.tar

where the connecting lines indicate hard links (i.e. same inode).

  • @sebix - example added; @muffel - should be easy to test the above with small tar files (add another small file to the tar between instances). rsync can give you stats about how much speedup is achieved, which should help you validate that the 'parent' file gets used as a seed for the 'current' file. – Toby Speight Jul 23 '15 at 18:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.