53

Is there something out there for parallel archiving of files?

Tar is great, but I don't use tape archives, and it's more important to me that the archiving happens quickly (with compression like bzip2) since I have smp.

Improve this question
4
  • 8
    tar is for more than just tapes. The name originally came from tape, but these days I see it being used mostly for when you want to put things into a single file for redistribution while maintaining directory structure information with optional compression.
    – Kevin M
    Commented Oct 11, 2010 at 14:48
  • there's quite a few parallel compression tools benchmarked here vbtechsupport.com/1614 however have yet to find a parallel version of tar
    – p4guru
    Commented May 21, 2015 at 2:31
  • None of the answer provided (including the accepted one) handle directories, so far as I can tell - they handle files. I only see zip as being able to handle directories :|
    – warren
    Commented Mar 3, 2016 at 17:23
  • 1
    Actually, what we usually do is to package directories in tar archives, and then compress the package using a file compressor (like gzip, pigz, etc.). You can do it in two steps, but also in one single step, since they can work on data streams from standard input/output. The results are very similar to zip, but more versatile.
    – gerlos
    Commented Nov 15, 2016 at 11:19

8 Answers 8

Reset to default
47

I think you are looking for pbzip2:

PBZIP2 is a parallel implementation of the bzip2 block-sorting file compressor that uses pthreads and achieves near-linear speedup on SMP machines.

Have a look at the project homepage or check your favorite package repository.

Improve this answer
4
  • 2
    You can also try pigz and pxz for parallel implementations of gzip and xz. You can compress using a command like tar c dir | pigz -c > dir.tar.gz and decompress using pigz -cd dir.tar.gz | tar xf -.
    – gerlos
    Commented Oct 29, 2015 at 19:31
  • 9
    Commands today would be tar -cf dir.tar.gz -I pigz dir and tar -xf dir.tar.gz -I pigz. Also xz is threaded: use XZ_OPT=-T0 tar -cJf dir.tar.gz dir and XZ_OPT=-T0 tar -xJf dir.tar.gz.
    – Rich
    Commented Aug 25, 2017 at 16:53
  • 1
    @Rich This question is not specific to any platform, meaning it applies to BSD and GNU alike. Your "today" version (which was just as usable 7 years earlier) only applies to GNU versions as -I means "read filenames from file" on BSD versions of tar (such as on macOS) and the XZ_OPT env var is unknown to the bsd version.
    – oligofren
    Commented Oct 23, 2021 at 4:45
  • Fair enough. I was writing specifically about GNU and not Unix.
    – Rich
    Commented Oct 29, 2021 at 17:46
28

The OP asked about parallel archiving, not parallel compression.

If the source material is coming from a filesystem where different directories/files might be on different disks, or even a single fast disk that exceeds the input speed of the compression tool(s), then could indeed be beneficial to have multiple streams of input going into the compression layers.

The meaningful question becomes, what does the output from a parallel archive look like? It's no longer just a single file descriptor / stdout, but a file descriptor per thread.

An example of this so far is the parallel dump mode of Postgresql pg_dump, wherein it dumps to a directory, with threads working over the set of tables to back up (work queue w/ multiple threads consuming the queue).

I'm not sure of any actual parallel archivers that are mainstream. There was a hack for Solaris Tar for use on ZFS: http://www.maier-komor.de/mtwrite.html

There are some dedicated backup tools that successfully run multiple threads, but lots more that just split the workload by directory at a high level.

Improve this answer
25

7zip can run on multiple threads when given the -mmt flag, but only when compressing into 7z-archives, which offer great compression but are generally slower than zip to create archives. Do something like this:

7z a -mmt foo.7z /opt/myhugefile.dat
Improve this answer
2
  • 1
    7z is a nice archiver, with good support for controlling the tradeoffs between compression ratio and comp/decomp time, random access vs. better compression, and stuff like that. However, it doesn't store nearly as much metadata as tar, you lose owner/permissions.
    – Peter Cordes
    Commented Aug 22, 2015 at 22:49
  • It looks like this options is on by default - at least I've got no performance increase with it and 7z' output has line about amount of cores of my CPU in both cases.
    – Andrey Starodubtsev
    Commented Oct 29, 2015 at 16:01
16 
tar --use-compress-program=pigz  ....

replace pigz with your favorite parallel compression program. The reason to use tar is because it can store the owner, group, permissions. That metadata is often useful (e.g., restoring a dir tree in a complex system).

Improve this answer
2
  • 6
    tar -c --use-compress-program=pigz -f myDirectory.tar.gz myDirectory/
    – markusN
    Commented Oct 28, 2015 at 16:22
  • 1
    All options, AFAIK, for tar can be used normally along with the -I option, which is the same as --use-compress-program. So, e.g. tar cvzf /some/dir/yournewarchive.tar.gz /directory/tobecompressed --exclude="/directo...." can be applied to the multi-threaded option using pigz as tar -I pigz -cvf /some/dir/yournewarchive.tar.gz /directory/tobecompressed --exclude="/directo...". This is the best, most appropriate answer, IMHO. Thanks @uDude ! :)
    – ILMostro_7
    Commented Nov 22, 2017 at 10:07
11

pigz is a parallel implementation of gzip, but can only really use multiple processors for compression, not decompression.

Improve this answer
3
  • 2
    Did some experiments, and pigz actually seems to be able to use multiple threads also when decompressing. Try comparing the output of time tar xf dir.tar.gz and of time pigz -cd dir.tar.gz | tar xf - (on my 4-core CPU it takes a bit less than half the time).
    – gerlos
    Commented Oct 29, 2015 at 19:21
  • 5
    @gerlos Using time on a pipeline will only time the first command. From the pigz documentation: "Decompression can't be parallelized, at least not without specially prepared deflate streams for that purpose. As a result, pigz uses a single thread (the main thread) for decompression, but will create three other threads for reading, writing, and check calculation, which can speed up decompression under some circumstances."
    – augurar
    Commented Aug 31, 2016 at 18:59
  • 1
    There's also pixz.
    – Marc.2377
    Commented Jun 22, 2017 at 3:22
8

tar is simply an archive format that is very good at exactly duplicating the files and preserving the directory tree and the original file attributes. TAR is very good for making backups, because everything is preserved. I use pbzip2 to compress the tar archives I use for system backups with very good results.

this command should do the trick.

tar -cpS "infile" | pbzip2 > "outfile"

pbzip2 can be replaced with a different compression utility, but be warned, LZMA compression (like pxz) uses a TON of RAM when compressing/decompressing large files (I tried to run 8 threads with 8GB of RAM, and pxz started swapping to disk).

Improve this answer
5

Another contender is lbzip2. It's quite similar to pbzip2

Improve this answer
5

As far as compression is considered, xz since about version 5.2 supports parallel compression via the -T option.

Improve this answer
1
  • 2
    If you want to use this in combination with tar just call export XZ_DEFAULTS="-T 0" before.
    – Étienne
    Commented Nov 5, 2020 at 15:41

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .