Sign up ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It's 100% free, no registration required.

Is there something out there for parallel archiving of files?

Tar is great, but I don't use tape archives, and it's more important to me that the archiving happens quickly (with compression like bzip2) since I have smp.

share|improve this question
tar is for more than just tapes. The name originally came from tape, but these days I see it being used mostly for when you want to put things into a single file for redistribution while maintaining directory structure information with optional compression. – Kevin M Oct 11 '10 at 14:48
there's quite a few parallel compression tools benchmarked here however have yet to find a parallel version of tar – p4guru May 21 at 2:31

6 Answers 6

up vote 25 down vote accepted

I think you are looking for pbzip2:

PBZIP2 is a parallel implementation of the bzip2 block-sorting file compressor that uses pthreads and achieves near-linear speedup on SMP machines.

Have a look at the project homepage or check your favorite package repository.

share|improve this answer
You can also try pigz and pxz for parallel implementations of gzip and xz. You can compress using a command like tar c dir | pigz -c > dir.tar.gz and decompress using pigz -cd dir.tar.gz | tar xf -. – gerlos Oct 29 at 19:31

7zip can run on multiple threads when given the -mmt flag, but only when compressing into 7z-archives which offer great compression but are generally slower to create then zip archives. Do something like this.

7z a -mmt foo.7z /opt/myhugefile.dat
share|improve this answer
7z is a nice archiver, with good support for controlling the tradeoffs between compression ratio and comp/decomp time, random access vs. better compression, and stuff like that. However, it doesn't store nearly as much metadata as tar, you lose owner/permissions. – Peter Cordes Aug 22 at 22:49
It looks like this options is on by default - at least I've got no performance increase with it and 7z' output has line about amount of cores of my CPU in both cases. – Andrey Starodubtsev Oct 29 at 16:01

pigz is a parallel implementation of gzip, but can only really use multiple processors for compression, not decompression.

share|improve this answer
Did some experiments, and pigz actually seems to be able to use multiple threads also when decompressing. Try comparing the output of time tar xf dir.tar.gz and of time pigz -cd dir.tar.gz | tar xf - (on my 4-core CPU it takes a bit less than half the time). – gerlos Oct 29 at 19:21

tar is simply an archive format that is very good at exactly duplicating the files and preserving the directory tree and the original file attributes. TAR is very good for making backups, because everything is preserved. I use pbzip2 to compress the tar archives I use for system backups with very good results.

this command should do the trick.

tar -cpS "infile" | pbzip2 > "outfile"

pbzip2 can be replaced with a different compression utility, but be warned, LZMA compression (like pxz) uses a TON of RAM when compressing/decompressing large files (I tried to run 8 threads with 8GB of RAM, and pxz started swapping to disk).

share|improve this answer

Another contender is lbzip2. It's quite similar to pbzip2

share|improve this answer
tar --use-compress-program=pigz  ....

replace pigz with your favorite parallel compression program. The reason to use tar is because it can store the owner, group, permissions. That metadata is often useful (e.g., restoring a dir tree in a complex system).

share|improve this answer
tar -c --use-compress-program=pigz -f myDirectory.tar.gz myDirectory/ – markusN Oct 28 at 16:22

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.