I need to create compressed archives of files, and be able to quickly extract individual files/directories from them.

The problem is, for example, tar.bz2 seems to be not the best choice for such task - extracting single 4kb file out of 200Mb archive (50000 files) takes 17 seconds on my machine.

Is there some archive format, that provides fast export of individual files from the archive (and works on linux)?


The Zip format compresses each file separately, and then combines them (with a directory of archive contents) into a single archive file.

  • Yes, zip format seems to be fast. What is the best util to compress/extract it under linux - I tried 7z, and compressing is 3x slower than bzip2. – Rogach Apr 15 '12 at 18:40
  • Info-ZIP is what everyone uses; disable compression entirely for maximum speed. – Ignacio Vazquez-Abrams Apr 15 '12 at 20:27
  • It doesn't actually compress each file separately. It chops the input files up into 32kb blocks and compresses each block separately. One block may only be part of a larger file, or it may contain several smaller files. – psusi Apr 16 '12 at 2:34

In addition to the already mentioned zip format, the dar and dump utilities also are good at handling this, and unlike zip, retain the unix permissions. For dar you want to avoid using the solid archive option, as that goes back to the tar/gzip method of compressing the whole thing at once, which gives better compression, but makes extracting individual files take longer as the whole file must be decompressed until the desired file is found. dump handles large sets of smallish files ( tens of thousands ) rather well, and can do multithreaded compression, but it only reads ext[234] filesystems.


pixz is a parallel, indexing version of xz.

# Compress:
tar -I pixz -cf foo.tar.xz ./foo

# Decompress:
tar -I pixz -xf foo.tar.xz

# Very quickly list the contents of the compressed tarball:
pixz -l foo.tar.xz

# Very quickly extract a single file:
pixz -x dir/file < foo.tar.xz | tar x

.tar.gz has a faster decompression speed, but the trade off is worst compression size. edit: Ignacio Vazquez-Abrams ninja'd me.

  • Yes, but will it read the whole archive file to extract the single small file? – Rogach Apr 15 '12 at 17:25
  • yes, that is a limitation of using tar itself. – llua Apr 15 '12 at 17:59
  • Sad. I need some archive type, that does not have that limitation :( – Rogach Apr 15 '12 at 18:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.