Compressing a folder but do not compress specific file types but include them in the gz file

Question

I am using tar with pigz to compress a folder and save a backup. Size of this folder ~250 GB or more. This folder has variety of content including numerous text and log files, ISOs and zip files in many different sub-folders. Full compression of this folder takes around 1 hour (or more sometimes). At this moment I use this in a script.

tar -cf - <data_folder> | pigz -1 > <output_file>.tar.$

I want to reduce the compression time by excluding compression on ISOs and zip files. I want them (ISOs and zip files) to be included in the gzip file as such (uncompressed).

My question is this: Is it possible to selectively compress files based on type, and still include uncompressed files in the gzip output? How to try this out?

Answer 1

No, you can't. At least not directly.

tar doesn't do any compression. It merely reads part of the (virtual) file system, and generates one cohesive stream from it. This stream is then often passed to a compression tool/library, for instance gzip/libz. The compression part does not see or even know about individual files. It just compresses the stream generated by tar. Therefore you cannot add selective compression to your current approach.

What you can do is incrementally build the tar archive, by compressing every file individually and then adding it to the tar archive. By doing so you can choose to add (for example) iso images uncompressed to the archive. Note however, that the tar archive itself will not be compressed. Consequently after untaring it, you would also have to uncompress each file individually, where appropriate.

How much time do you actually lose by compressing isos and zip files? Seeing as tar | pigz > <file> is stream processing I'd guess you are not loosing that much time. There are blocks written to disk, while the next blocks are being compressed, while the stream is being built. It is happening in parallel.

Maybe you can optimize your strategy:

You could put all iso and zip files into dedicated directories and then build your archive in three steps: tar&compress the rest, add iso dir, add zip dir. The resulting archive still needs a lengthy extraction procedure of untaring the outer archive and then uncompressing and untaring the inner archive. Yet, this is more feasible than uncompressing every individual file.

Or you tune the commands: Does it have to be a tar archive of a file system or could you use dd to backup the entire partition? Backing up the entire partition has the advantage of continuous reads from the disk(s) which may be faster than working with the file system. I am sure you can tune pigz to work with bigger chunks, which should give you a speed up, if iso and zip files are your problem. Also, you could add some buffering (e.g. mbuffer), before writing the result to disk to further optimize media access.