9

I need to create compressed archives of files, and be able to quickly extract individual files/directories from them.

The problem is, for example, tar.bz2 seems to be not the best choice for such task - extracting single 4kb file out of 200Mb archive (50000 files) takes 17 seconds on my machine.

Is there some archive format, that provides fast export of individual files from the archive (and works on linux)?

Improve this question

5 Answers 5

Reset to default
6

pixz is a parallel, indexing version of xz.

# Compress:
tar -I pixz -cf foo.tar.xz ./foo

# Decompress:
tar -I pixz -xf foo.tar.xz

# Very quickly list the contents of the compressed tarball:
pixz -l foo.tar.xz

# Very quickly extract a single file:
pixz -x dir/file < foo.tar.xz | tar x
Improve this answer
5

The Zip format compresses each file separately, and then combines them (with a directory of archive contents) into a single archive file.

Improve this answer
3
  • Yes, zip format seems to be fast. What is the best util to compress/extract it under linux - I tried 7z, and compressing is 3x slower than bzip2.
    – Rogach
    Commented Apr 15, 2012 at 18:40
  • Info-ZIP is what everyone uses; disable compression entirely for maximum speed.
    – Ignacio Vazquez-Abrams
    Commented Apr 15, 2012 at 20:27
  • 1
    It doesn't actually compress each file separately. It chops the input files up into 32kb blocks and compresses each block separately. One block may only be part of a larger file, or it may contain several smaller files.
    – psusi
    Commented Apr 16, 2012 at 2:34
4

In addition to the already mentioned zip format, the dar and dump utilities also are good at handling this, and unlike zip, retain the unix permissions. For dar you want to avoid using the solid archive option, as that goes back to the tar/gzip method of compressing the whole thing at once, which gives better compression, but makes extracting individual files take longer as the whole file must be decompressed until the desired file is found. dump handles large sets of smallish files ( tens of thousands ) rather well, and can do multithreaded compression, but it only reads ext[234] filesystems.

Improve this answer
2

Squashfs is best for random access

Improve this answer
0

.tar.gz has a faster decompression speed, but the trade off is worst compression size. edit: Ignacio Vazquez-Abrams ninja'd me.

Improve this answer
3
  • Yes, but will it read the whole archive file to extract the single small file?
    – Rogach
    Commented Apr 15, 2012 at 17:25
  • 1
    yes, that is a limitation of using tar itself.
    – llua
    Commented Apr 15, 2012 at 17:59
  • Sad. I need some archive type, that does not have that limitation :(
    – Rogach
    Commented Apr 15, 2012 at 18:09

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .