109

I need to backup a fairly large directory, but I am limited by the size of individual files. I'd like to essentially create a tar.(gz|bz2) archive which is split into 200MB maximum archives. Clonezilla does something similar to this by splitting image backups named like so:

sda1.backup.tar.gz.aa
sda1.backup.tar.gz.ab
sda1.backup.tar.gz.ac

Is there a way I can do this in one command? I understand how to use the split command, but I'd like to not have to create one giant archive, then split it into smaller archives, as this would double the disk space I'd need in order to initially create the archive.

1

7 Answers 7

143

You can pipe tar to the split command:

tar cvzf - dir/ | split --bytes=200MB - sda1.backup.tar.gz.

On some *nix systems (like OS X) you may get the following error:

split: illegal option -- -

In that case try this (note the -b 200m):

tar cvzf - dir/ | split -b 200m - sda1.backup.tar.gz.

If you happen to be trying to split the file to fit on a FAT32 formatted drive, use a byte limit of 4294967295. For example:

tar cvzf - /Applications/Install\ macOS\ Sierra.app/ | \
split -b 4294967295 - /Volumes/UNTITLED/install_macos_sierra.tgz.

When you want to extract the files use the following command (as of @Naftuli Kay commented):

cat sda1.backup.tar.gz.* | tar xzvf -
7
  • 27
    Will cat sda1.backup.tar.gz.* | tar xzvf - do the job? Jan 18, 2013 at 19:45
  • 4
    Yes, it should. split by default sets the names the files so that when sorted by LOCALE (which is done by shell globbing) will be in the correct order.
    – jordanm
    Jan 18, 2013 at 19:47
  • @NaftuliTzviKay Using cat on Command line works fine. But when I do the same using a she'll Script, I am thrown at error saying the file.tar.gz.* not found.
    – Vinay
    Dec 8, 2014 at 17:33
  • 4
    Without verbose, just do tar czf ... without the v and merge by cat backup.tar.gz.* | tar tar xzf - without v. I see no benefit of the verbose output here by v. Jul 9, 2016 at 12:15
  • 1
    Just helped a friend by packing Xcode onto a FAT32 formatted flash drive with: tar cvzf - Xcode.app/ | split -b 2000m - /Volumes/PH/xcode/xcode.tgz (used from cd /Applications/) Thank you very much :)
    – ecth
    Dec 8, 2017 at 12:06
15

On macOS, the split command works slightly differently:

$ tar cvzf - foo | split -b 2500m - foo.tgz.
5

tar split archive

I found this to be the best solution for a few reasons:

  • It creates parts without interaction, automatically naming parts
  • You can use any compression you want, usual tar options
  • Requires no external commands for splitting or joining
  • Uses no extra disk space (intermediate)
  • Any dearchiver handles easily as each archive is self-contained
  • Increase safety as each archive is self-contained, files do not span multiple archives

This command is creating 2GB chunks without the compression:

tar -cv --tape-length=2097000 --file=my_archive-{0..50}.tar file1 file2 dir3
  • c for create
  • v for verbose, to list files added to the archive
  • --tape-length is chunk size: you can add a suffix, if you omit it, a kilobyte is assumed (hence 2 million for a 2 gigabyte)
  • --file is where we magically create names for chunks: we give arbitrarily 50 but you may put any big enough number, only those needed will be used
  • list of files and directories to be included in archives

Similarily, this command is creating 1GB chunks with the gzip compression:

tar -czv --tape-length=2097000 --file=my_archive-{0..50}.tar.gz file1 file2 dir3
2
  • Note that (I'm pretty sure) the tar file won't be broken at file boundaries, meaning half a file can be in one tar archive, and the other half is in the next tar archive. At least, that seems to be the case from the errors I'm seeing when trying to extract a single tar archive ("Unexpected EOF in archive"). Just mentioning this to help others in my situation. Please correct me if I'm wrong.
    – joe
    Feb 23 at 13:35
  • I would use --file=my_archive.tar.gz.{00..50} instead, for two reasons: First, placing the number at the end indicates that the file is just a part of a larger archive. Second, using fixed-width numbers will sort the files correctly when using cat to recombine the pieces. Apr 1 at 17:10
4
serega@serega-sv:~$ tar -c  -M --tape-length=1024 --file /tmp/pseudo-tape.tar --new-volume-script=/tmp/new-volume.sh --volno-file=/tmp/volno /tmp/stuff-to-archive 
tar: Removing leading `/' from member names
moving /tmp/pseudo-tape.tar to /tmp/archive.1
moving /tmp/pseudo-tape.tar to /tmp/archive.2
moving /tmp/pseudo-tape.tar to /tmp/archive.3

You'll need a script for automation moving pseudo-tape.tar file to a new name:

serega@serega-sv:~$ cat /tmp/new-volume.sh 
dir="/tmp"
base_name="pseudo-tape.tar"
next_volume_name=`echo -n "archive."; cat $dir/volno`
echo "moving $dir/$base_name to $dir/$next_volume_name"
mv "$dir/$base_name" "$dir/$next_volume_name"
2
  • I haven't downvoted your answer, because I am happy to see one that uses -M --tape-length. However, this answer does ignore the OP's request for a solution that uses gzip or bzip2 compression.
    – user6860
    Mar 15, 2018 at 13:28
  • 2
    Caution: you are not talking about tar but rather about a tar clone called gtar (GNU tar). This tar clone supports to create multi volume archives but with a noticeable probability is unable/unwilling to extract from those multi volume archives as it incorrectly claims that a follow up volume is not the right continuation part.
    – schily
    Sep 1, 2018 at 11:32
4

Just to add: As the maximum allowed file size in vfat/fat32 is 2^32 minus 1 (4294967295 bytes), the split command with the maximum allowed file size on such file system is:

split -b4294967295 -d my_input_file my_output_file_splitted
2

Just to throw in my own contribution, I wrote an app recently that splits up tarballs along file boundaries, which you may find useful:

https://github.com/dmuth/tarsplit

1

Instead of tar I'd use 7zip or some other archiver that can natively split archive of file boundaries.

With split command you may have roubles recovering faulty archives when just one part of the series gets damaged.

7z and some other archives additionally may create recovery sum added to archives or even have option to add recovery volume that saves your day when you loose or damage entire part.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.