102

I need to backup a fairly large directory, but I am limited by the size of individual files. I'd like to essentially create a tar.(gz|bz2) archive which is split into 200MB maximum archives. Clonezilla does something similar to this by splitting image backups named like so:

sda1.backup.tar.gz.aa
sda1.backup.tar.gz.ab
sda1.backup.tar.gz.ac

Is there a way I can do this in one command? I understand how to use the split command, but I'd like to not have to create one giant archive, then split it into smaller archives, as this would double the disk space I'd need in order to initially create the archive.

1
139

You can pipe tar to the split command:

tar cvzf - dir/ | split --bytes=200MB - sda1.backup.tar.gz.

On some *nix systems (like OS X) you may get the following error:

split: illegal option -- -

In that case try this (note the -b 200m):

tar cvzf - dir/ | split -b 200m - sda1.backup.tar.gz.

If you happen to be trying to split the file to fit on a FAT32 formatted drive, use a byte limit of 4294967295. For example:

tar cvzf - /Applications/Install\ macOS\ Sierra.app/ | \
split -b 4294967295 - /Volumes/UNTITLED/install_macos_sierra.tgz.

When you want to extract the files use the following command (as of @Naftuli Kay commented):

cat sda1.backup.tar.gz.* | tar xzvf -
7
  • 27
    Will cat sda1.backup.tar.gz.* | tar xzvf - do the job? Jan 18 '13 at 19:45
  • 4
    Yes, it should. split by default sets the names the files so that when sorted by LOCALE (which is done by shell globbing) will be in the correct order.
    – jordanm
    Jan 18 '13 at 19:47
  • @NaftuliTzviKay Using cat on Command line works fine. But when I do the same using a she'll Script, I am thrown at error saying the file.tar.gz.* not found.
    – Vinay
    Dec 8 '14 at 17:33
  • 4
    Without verbose, just do tar czf ... without the v and merge by cat backup.tar.gz.* | tar tar xzf - without v. I see no benefit of the verbose output here by v. Jul 9 '16 at 12:15
  • 1
    Just helped a friend by packing Xcode onto a FAT32 formatted flash drive with: tar cvzf - Xcode.app/ | split -b 2000m - /Volumes/PH/xcode/xcode.tgz (used from cd /Applications/) Thank you very much :)
    – ecth
    Dec 8 '17 at 12:06
14

On macOS, the split command works slightly differently:

$ tar cvzf - foo | split -b 2500m - foo.tgz.
4
serega@serega-sv:~$ tar -c  -M --tape-length=1024 --file /tmp/pseudo-tape.tar --new-volume-script=/tmp/new-volume.sh --volno-file=/tmp/volno /tmp/stuff-to-archive 
tar: Removing leading `/' from member names
moving /tmp/pseudo-tape.tar to /tmp/archive.1
moving /tmp/pseudo-tape.tar to /tmp/archive.2
moving /tmp/pseudo-tape.tar to /tmp/archive.3

You'll need a script for automation moving pseudo-tape.tar file to a new name:

serega@serega-sv:~$ cat /tmp/new-volume.sh 
dir="/tmp"
base_name="pseudo-tape.tar"
next_volume_name=`echo -n "archive."; cat $dir/volno`
echo "moving $dir/$base_name to $dir/$next_volume_name"
mv "$dir/$base_name" "$dir/$next_volume_name"
2
  • I haven't downvoted your answer, because I am happy to see one that uses -M --tape-length. However, this answer does ignore the OP's request for a solution that uses gzip or bzip2 compression.
    – user6860
    Mar 15 '18 at 13:28
  • 2
    Caution: you are not talking about tar but rather about a tar clone called gtar (GNU tar). This tar clone supports to create multi volume archives but with a noticeable probability is unable/unwilling to extract from those multi volume archives as it incorrectly claims that a follow up volume is not the right continuation part.
    – schily
    Sep 1 '18 at 11:32
4

Just to add: As the maximum allowed file size in vfat/fat32 is 2^32 minus 1 (4294967295 bytes), the split command with the maximum allowed file size on such file system is:

split -b4294967295 -d my_input_file my_output_file_splitted
1

Just to throw in my own contribution, I wrote an app recently that splits up tarballs along file boundaries, which you may find useful:

https://github.com/dmuth/tarsplit

0

Instead of tar I'd use 7zip or some other archiver that can natively split archive of file boundaries.

With split command you may have roubles recovering faulty archives when just one part of the series gets damaged.

7z and some other archives additionally may create recovery sum added to archives or even have option to add recovery volume that saves your day when you loose or damage entire part.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.