1

Objective: The efficient backup and transfer of ~/dircontaininginnumeraltinyfiles/ to /mnt/server/backups/dir.tgz. My first thought was of course rsync and gzip. But I'm open to other routes as well.

(At the end of the desired backup process, /mnt/server/backups/dir.tgz would be a full backup i.e. containing all of the files, not just ones changed since last backup. In a nutshell, I'm simply looking for a solution that's more efficient on the compression and transfer steps than tar -cvzf /mnt/server/backups/dir.tgz ~/localdir/.)

The local creation of any files is undesirable (e.g. a local .tgz backup and subsequent sync to the server) and instead have anything local just in memory e.g. via piping.

To clarify, the reason I don't want to simply rsync the dir to the local network server is because the source directory contains innumerable, highly compressible, tiny files. So for backup purposes a single, acutely overall smaller .tgz file is quite attractive.

That said, the significant majority of the files are unchanged per backup, so a simple tar -cvzf /destination/blah.tgz ~/sourcedir/ is rather inefficient, hence the desire for a smart, delta-only sort of process re the compression aspect.

While the amount of data isn't overbearing for 1Gb local network, some only have a 100Mb connection, hence the desire for a smart, delta-only sort of process for the transfer aspect as well would be desirable.

As a side note, one aspect I'm right now doing homework on is tar's --listed-incremental option and gzip's --rsyncable option.

7
  • When editing your original question, please exert caution, for already existing answers may become either wholly inadequate or without focus in the light of your edited post. Other users who may browse this later will find answered questions with incoate answers, that is questions not matching answers. So well-intended changes in an initial post may actually muddle the logic of already existing answers. You may edit your question to change its focus, but do so by adding an Edit: section at the end of your post. Cheers.
    – Cbhihe
    May 29, 2020 at 8:25
  • 1
    @Cbhihe no, if the edit is so different, then it should be a new question and not an edit. We really, really don't want Edit sections, ever.
    – terdon
    May 29, 2020 at 16:56
  • 1
    I don't yet see a drastic change in scope for this question. I'll just note a bit of shared responsibility between askers and answerers. If a question is not yet clear, resist the temptation to answer it; instead, leave comments to nudge the question into clarity. Once a clear question has been answered, the question should not be modified in a way that invalidates any existing answers.
    – Jeff Schaller
    May 29, 2020 at 18:54
  • 1
    @JeffSchaller: duly noted.
    – Cbhihe
    May 29, 2020 at 19:07
  • 1
    @Cbhihe sorry, didn't mean to suggest you did anything wrong, I know you were trying to help! For future reference though, if a question has been answered and the answer leads to a different question, it is better to post a new question rather than have "edit" sections. But yeah, not really a big deal one way or the other :)
    – terdon
    May 29, 2020 at 19:46

1 Answer 1

0

Using tar -cvzf /mnt/server/backups/dir.tgz ~/localdir/ for the time being. Respectfully stepping away from this thread. Thanks for the input. Best regards to all.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .