2

I am using tar with pigz to compress a folder and save a backup. Size of this folder ~250 GB or more. This folder has variety of content including numerous text and log files, ISOs and zip files in many different sub-folders. Full compression of this folder takes around 1 hour (or more sometimes). At this moment I use this in a script.

tar -cf - <data_folder> | pigz -1 > <output_file>.tar.$

I want to reduce the compression time by excluding compression on ISOs and zip files. I want them (ISOs and zip files) to be included in the gzip file as such (uncompressed).

My question is this: Is it possible to selectively compress files based on type, and still include uncompressed files in the gzip output? How to try this out?

  • What you are currently doing is a tar archive that you compress via a pipe. All that pipe compressor sees is raw data, not files. It would be a much more tedious and less reliable process to compress files and then add them to the archive, and would also compress less. – Julie Pelletier May 20 '16 at 7:18
  • Did you try to increase the block size for pigz (maybe with -b 1024) to lower the overhead and probably increase speed? 250 GB gives a lot of 128 KiB chunks. – Dubu May 20 '16 at 7:27
4

No, you can't. At least not directly.

tar doesn't do any compression. It merely reads part of the (virtual) file system, and generates one cohesive stream from it. This stream is then often passed to a compression tool/library, for instance gzip/libz. The compression part does not see or even know about individual files. It just compresses the stream generated by tar. Therefore you cannot add selective compression to your current approach.

What you can do is incrementally build the tar archive, by compressing every file individually and then adding it to the tar archive. By doing so you can choose to add (for example) iso images uncompressed to the archive. Note however, that the tar archive itself will not be compressed. Consequently after untaring it, you would also have to uncompress each file individually, where appropriate.

How much time do you actually lose by compressing isos and zip files? Seeing as tar | pigz > <file> is stream processing I'd guess you are not loosing that much time. There are blocks written to disk, while the next blocks are being compressed, while the stream is being built. It is happening in parallel.

Maybe you can optimize your strategy:

You could put all iso and zip files into dedicated directories and then build your archive in three steps: tar&compress the rest, add iso dir, add zip dir. The resulting archive still needs a lengthy extraction procedure of untaring the outer archive and then uncompressing and untaring the inner archive. Yet, this is more feasible than uncompressing every individual file.

Or you tune the commands: Does it have to be a tar archive of a file system or could you use dd to backup the entire partition? Backing up the entire partition has the advantage of continuous reads from the disk(s) which may be faster than working with the file system. I am sure you can tune pigz to work with bigger chunks, which should give you a speed up, if iso and zip files are your problem. Also, you could add some buffering (e.g. mbuffer), before writing the result to disk to further optimize media access.

  • Thank you for your suggestions. Your first strategy seems feasible and I think I can restructure the contents in three folders and then compress only the one (which does not have iso and zips). – ankit7540 May 20 '16 at 11:27
  • By the way, I have a question, can dd be used with pigz ? – ankit7540 May 20 '16 at 11:28
  • 1
    @ankit7540 please ask your new question as a new question. – Bananguin Jun 22 '17 at 21:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.