Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I need to create compressed archives of files, and be able to quickly extract individual files/directories from them.

The problem is, for example, tar.bz2 seems to be not the best choice for such task - extracting single 4kb file out of 200Mb archive (50000 files) takes 17 seconds on my machine.

Is there some archive format, that provides fast export of individual files from the archive (and works on linux)?

share|improve this question

3 Answers 3

up vote 4 down vote accepted

The Zip format compresses each file separately, and then combines them (with a directory of archive contents) into a single archive file.

share|improve this answer
    
Yes, zip format seems to be fast. What is the best util to compress/extract it under linux - I tried 7z, and compressing is 3x slower than bzip2. –  Rogach Apr 15 '12 at 18:40
    
Info-ZIP is what everyone uses; disable compression entirely for maximum speed. –  Ignacio Vazquez-Abrams Apr 15 '12 at 20:27
    
It doesn't actually compress each file separately. It chops the input files up into 32kb blocks and compresses each block separately. One block may only be part of a larger file, or it may contain several smaller files. –  psusi Apr 16 '12 at 2:34

In addition to the already mentioned zip format, the dar and dump utilities also are good at handling this, and unlike zip, retain the unix permissions. For dar you want to avoid using the solid archive option, as that goes back to the tar/gzip method of compressing the whole thing at once, which gives better compression, but makes extracting individual files take longer as the whole file must be decompressed until the desired file is found. dump handles large sets of smallish files ( tens of thousands ) rather well, and can do multithreaded compression, but it only reads ext[234] filesystems.

share|improve this answer

.tar.gz has a faster decompression speed, but the trade off is worst compression size. edit: Ignacio Vazquez-Abrams ninja'd me.

share|improve this answer
    
Yes, but will it read the whole archive file to extract the single small file? –  Rogach Apr 15 '12 at 17:25
    
yes, that is a limitation of using tar itself. –  llua Apr 15 '12 at 17:59
    
Sad. I need some archive type, that does not have that limitation :( –  Rogach Apr 15 '12 at 18:09

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.