Objective: The efficient backup and transfer of ~/dircontaininginnumeraltinyfiles/ to /mnt/server/backups/dir.tgz. My first thought was of course rsync and gzip. But I'm open to other routes as well.
(At the end of the desired backup process, /mnt/server/backups/dir.tgz would be a full backup i.e. containing all of the files, not just ones changed since last backup. In a nutshell, I'm simply looking for a solution that's more efficient on the compression and transfer steps than tar -cvzf /mnt/server/backups/dir.tgz ~/localdir/.)
The local creation of any files is undesirable (e.g. a local .tgz backup and subsequent sync to the server) and instead have anything local just in memory e.g. via piping.
To clarify, the reason I don't want to simply rsync the dir to the local network server is because the source directory contains innumerable, highly compressible, tiny files. So for backup purposes a single, acutely overall smaller .tgz file is quite attractive.
That said, the significant majority of the files are unchanged per backup, so a simple tar -cvzf /destination/blah.tgz ~/sourcedir/ is rather inefficient, hence the desire for a smart, delta-only sort of process re the compression aspect.
While the amount of data isn't overbearing for 1Gb local network, some only have a 100Mb connection, hence the desire for a smart, delta-only sort of process for the transfer aspect as well would be desirable.
As a side note, one aspect I'm right now doing homework on is tar's --listed-incremental option and gzip's --rsyncable option.
