40

Is there something out there for parallel archiving of files?

Tar is great, but I don't use tape archives, and it's more important to me that the archiving happens quickly (with compression like bzip2) since I have smp.

  • 7
    tar is for more than just tapes. The name originally came from tape, but these days I see it being used mostly for when you want to put things into a single file for redistribution while maintaining directory structure information with optional compression. – Kevin M Oct 11 '10 at 14:48
  • there's quite a few parallel compression tools benchmarked here vbtechsupport.com/1614 however have yet to find a parallel version of tar – p4guru May 21 '15 at 2:31
  • None of the answer provided (including the accepted one) handle directories, so far as I can tell - they handle files. I only see zip as being able to handle directories :| – warren Mar 3 '16 at 17:23
  • 1
    Actually, what we usually do is to package directories in tar archives, and then compress the package using a file compressor (like gzip, pigz, etc.). You can do it in two steps, but also in one single step, since they can work on data streams from standard input/output. The results are very similar to zip, but more versatile. – gerlos Nov 15 '16 at 11:19
36

I think you are looking for pbzip2:

PBZIP2 is a parallel implementation of the bzip2 block-sorting file compressor that uses pthreads and achieves near-linear speedup on SMP machines.

Have a look at the project homepage or check your favorite package repository.

  • 1
    You can also try pigz and pxz for parallel implementations of gzip and xz. You can compress using a command like tar c dir | pigz -c > dir.tar.gz and decompress using pigz -cd dir.tar.gz | tar xf -. – gerlos Oct 29 '15 at 19:31
  • 3
    Commands today would be tar -cf dir.tar.gz -I pigz dir and tar -xf dir.tar.gz -I pigz. Also xz is threaded: use XZ_OPT=-T0 tar -cJf dir.tar.gz dir and XZ_OPT=-T0 tar -xJf dir.tar.gz. – Rich Aug 25 '17 at 16:53
21

7zip can run on multiple threads when given the -mmt flag, but only when compressing into 7z-archives, which offer great compression but are generally slower than zip to create archives. Do something like this:

7z a -mmt foo.7z /opt/myhugefile.dat
  • 7z is a nice archiver, with good support for controlling the tradeoffs between compression ratio and comp/decomp time, random access vs. better compression, and stuff like that. However, it doesn't store nearly as much metadata as tar, you lose owner/permissions. – Peter Cordes Aug 22 '15 at 22:49
  • It looks like this options is on by default - at least I've got no performance increase with it and 7z' output has line about amount of cores of my CPU in both cases. – Andrey Starodubtsev Oct 29 '15 at 16:01
14

The OP asked about parallel archiving, not parallel compression.

If the source material is coming from a filesystem where different directories/files might be on different disks, or even a single fast disk that exceeds the input speed of the compression tool(s), then could indeed be beneficial to have multiple streams of input going into the compression layers.

The meaningful question becomes, what does the output from a parallel archive look like? It's no longer just a single file descriptor / stdout, but a file descriptor per thread.

An example of this so far is the parallel dump mode of Postgresql pg_dump, wherein it dumps to a directory, with threads working over the set of tables to back up (work queue w/ multiple threads consuming the queue).

I'm not sure of any actual parallel archivers that are mainstream. There was a hack for Solaris Tar for use on ZFS: http://www.maier-komor.de/mtwrite.html

There are some dedicated backup tools that successfully run multiple threads, but lots more that just split the workload by directory at a high level.

11
tar --use-compress-program=pigz  ....

replace pigz with your favorite parallel compression program. The reason to use tar is because it can store the owner, group, permissions. That metadata is often useful (e.g., restoring a dir tree in a complex system).

  • 4
    tar -c --use-compress-program=pigz -f myDirectory.tar.gz myDirectory/ – markusN Oct 28 '15 at 16:22
  • 1
    All options, AFAIK, for tar can be used normally along with the -I option, which is the same as --use-compress-program. So, e.g. tar cvzf /some/dir/yournewarchive.tar.gz /directory/tobecompressed --exclude="/directo...." can be applied to the multi-threaded option using pigz as tar -I pigz -cvf /some/dir/yournewarchive.tar.gz /directory/tobecompressed --exclude="/directo...". This is the best, most appropriate answer, IMHO. Thanks @uDude ! :) – ILMostro_7 Nov 22 '17 at 10:07
10

pigz is a parallel implementation of gzip, but can only really use multiple processors for compression, not decompression.

  • 2
    Did some experiments, and pigz actually seems to be able to use multiple threads also when decompressing. Try comparing the output of time tar xf dir.tar.gz and of time pigz -cd dir.tar.gz | tar xf - (on my 4-core CPU it takes a bit less than half the time). – gerlos Oct 29 '15 at 19:21
  • 4
    @gerlos Using time on a pipeline will only time the first command. From the pigz documentation: "Decompression can't be parallelized, at least not without specially prepared deflate streams for that purpose. As a result, pigz uses a single thread (the main thread) for decompression, but will create three other threads for reading, writing, and check calculation, which can speed up decompression under some circumstances." – augurar Aug 31 '16 at 18:59
  • 1
    There's also pixz. – Marc.2377 Jun 22 '17 at 3:22
8

tar is simply an archive format that is very good at exactly duplicating the files and preserving the directory tree and the original file attributes. TAR is very good for making backups, because everything is preserved. I use pbzip2 to compress the tar archives I use for system backups with very good results.

this command should do the trick.

tar -cpS "infile" | pbzip2 > "outfile"

pbzip2 can be replaced with a different compression utility, but be warned, LZMA compression (like pxz) uses a TON of RAM when compressing/decompressing large files (I tried to run 8 threads with 8GB of RAM, and pxz started swapping to disk).

4

Another contender is lbzip2. It's quite similar to pbzip2

1

As far as compression is considered, xz since about version 5.2 supports parallel compression via the -T option.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.