I've got a file.tar.gz that I create and backup to Amazon S3. It's gotten bigger than Amazon's limits and thus has not been backing up.

This is my current command:

tar -cpzf file.tar.gz directory/

Is it better, with TAR's --exclude functionality to skip a directory where the bulk of files live or TAR and Gzip the entire directory and then use something like 'split' to break it up into smaller chunks?

  • 1
    What about using things like duplicity? Commented Mar 10, 2014 at 21:01
  • 1
    What is the limit imposed? Maximum file size or maximum disk usage? What do you mean by "better"? In terms of what? Speed? Ease of use?
    – terdon
    Commented Mar 10, 2014 at 21:16
  • I've just started looking at duplicity. That may be a better solution than my homegrown script. The limit at Amazon is 4GB. My file, which is a backup of web uploads, has grown to over 6GB now. Incremental backups with something like duplicity may be a better strategy. Commented Mar 10, 2014 at 21:18

2 Answers 2


If you are splitting, split according the directory structure. Much more portable and much easier to decompress. It's easy to break a multiply split file (it's all interdependent). On the other hand, if you have 10 archives instead of one, you can just untar all at once with globbing. No harder than one file.


xz compresses even better than bzip2, but (again) not by very much.

You should look into just saving the files that have changed, or that aren't included e.g. in (re)installable packages.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .