7

I need to create compressed archives of files, and be able to quickly extract individual files/directories from them.

The problem is, for example, tar.bz2 seems to be not the best choice for such task - extracting single 4kb file out of 200Mb archive (50000 files) takes 17 seconds on my machine.

Is there some archive format, that provides fast export of individual files from the archive (and works on linux)?

4 Answers 4

4

The Zip format compresses each file separately, and then combines them (with a directory of archive contents) into a single archive file.

3
  • Yes, zip format seems to be fast. What is the best util to compress/extract it under linux - I tried 7z, and compressing is 3x slower than bzip2.
    – Rogach
    Apr 15, 2012 at 18:40
  • Info-ZIP is what everyone uses; disable compression entirely for maximum speed. Apr 15, 2012 at 20:27
  • 1
    It doesn't actually compress each file separately. It chops the input files up into 32kb blocks and compresses each block separately. One block may only be part of a larger file, or it may contain several smaller files.
    – psusi
    Apr 16, 2012 at 2:34
3

In addition to the already mentioned zip format, the dar and dump utilities also are good at handling this, and unlike zip, retain the unix permissions. For dar you want to avoid using the solid archive option, as that goes back to the tar/gzip method of compressing the whole thing at once, which gives better compression, but makes extracting individual files take longer as the whole file must be decompressed until the desired file is found. dump handles large sets of smallish files ( tens of thousands ) rather well, and can do multithreaded compression, but it only reads ext[234] filesystems.

2

pixz is a parallel, indexing version of xz.

# Compress:
tar -I pixz -cf foo.tar.xz ./foo

# Decompress:
tar -I pixz -xf foo.tar.xz

# Very quickly list the contents of the compressed tarball:
pixz -l foo.tar.xz

# Very quickly extract a single file:
pixz -x dir/file < foo.tar.xz | tar x
1

.tar.gz has a faster decompression speed, but the trade off is worst compression size. edit: Ignacio Vazquez-Abrams ninja'd me.

3
  • Yes, but will it read the whole archive file to extract the single small file?
    – Rogach
    Apr 15, 2012 at 17:25
  • 1
    yes, that is a limitation of using tar itself.
    – llua
    Apr 15, 2012 at 17:59
  • Sad. I need some archive type, that does not have that limitation :(
    – Rogach
    Apr 15, 2012 at 18:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.