Create a tar archive split into blocks of a maximum size

Question

I need to backup a fairly large directory, but I am limited by the size of individual files. I'd like to essentially create a tar.(gz|bz2) archive which is split into 200MB maximum archives. Clonezilla does something similar to this by splitting image backups named like so:

sda1.backup.tar.gz.aa
sda1.backup.tar.gz.ab
sda1.backup.tar.gz.ac

Is there a way I can do this in one command? I understand how to use the split command, but I'd like to not have to create one giant archive, then split it into smaller archives, as this would double the disk space I'd need in order to initially create the archive.

Answer 1

You can pipe tar to the split command:

tar cvzf - dir/ | split --bytes=200MB - sda1.backup.tar.gz.

On some *nix systems (like OS X) you may get the following error:

split: illegal option -- -

In that case try this (note the -b 200m):

tar cvzf - dir/ | split -b 200m - sda1.backup.tar.gz.

If you happen to be trying to split the file to fit on a FAT32 formatted drive, use a byte limit of 4294967295. For example:

tar cvzf - /Applications/Install\ macOS\ Sierra.app/ | \
split -b 4294967295 - /Volumes/UNTITLED/install_macos_sierra.tgz.

When you want to extract the files use the following command (as of @Naftuli Kay commented):

cat sda1.backup.tar.gz.* | tar xzvf -

Answer 2

On macOS, the split command works slightly differently:

$ tar cvzf - foo | split -b 2500m - foo.tgz.

Answer 3

tar split archive

I found this to be the best solution for a few reasons:

• It creates parts without interaction, automatically naming parts
• You can use any compression you want, usual tar options
• Requires no external commands for splitting or joining
• Uses no extra disk space (intermediate)
• Any dearchiver handles easily as each archive is self-contained
• Increase safety as each archive is self-contained, files do not span multiple archives

This command is creating 2GB chunks without the compression:

tar -cv --tape-length=2097000 --file=my_archive-{0..50}.tar file1 file2 dir3

• c for create
• v for verbose, to list files added to the archive
• --tape-length is chunk size: you can add a suffix, if you omit it, a kilobyte is assumed (hence 2 million for a 2 gigabyte)
• --file is where we magically create names for chunks: we give arbitrarily 50 but you may put any big enough number, only those needed will be used
• list of files and directories to be included in archives

Similarily, this command is creating 1GB chunks with the gzip compression:

tar -czv --tape-length=2097000 --file=my_archive-{0..50}.tar.gz file1 file2 dir3

Answer 4

Just to add: As the maximum allowed file size in vfat/fat32 is 2^32 minus 1 (4294967295 bytes), the split command with the maximum allowed file size on such file system is:

split -b4294967295 -d my_input_file my_output_file_splitted

Answer 5

serega@serega-sv:~$ tar -c  -M --tape-length=1024 --file /tmp/pseudo-tape.tar --new-volume-script=/tmp/new-volume.sh --volno-file=/tmp/volno /tmp/stuff-to-archive 
tar: Removing leading `/' from member names
moving /tmp/pseudo-tape.tar to /tmp/archive.1
moving /tmp/pseudo-tape.tar to /tmp/archive.2
moving /tmp/pseudo-tape.tar to /tmp/archive.3

You'll need a script for automation moving pseudo-tape.tar file to a new name:

serega@serega-sv:~$ cat /tmp/new-volume.sh 
dir="/tmp"
base_name="pseudo-tape.tar"
next_volume_name=`echo -n "archive."; cat $dir/volno`
echo "moving $dir/$base_name to $dir/$next_volume_name"
mv "$dir/$base_name" "$dir/$next_volume_name"

Answer 6

Just to throw in my own contribution, I wrote an app recently that splits up tarballs along file boundaries, which you may find useful:

https://github.com/dmuth/tarsplit

Answer 7

Instead of tar I'd use 7zip or some other archiver that can natively split archive of file boundaries.

With split command you may have roubles recovering faulty archives when just one part of the series gets damaged.

7z and some other archives additionally may create recovery sum added to archives or even have option to add recovery volume that saves your day when you loose or damage entire part.