I've got a tree of directories with files like this:

|-- 00
|   |-- 00
|   |   |-- 00
|   |   |-- 01
|   |   |-- 02
|   |   |-- 03
|   |   |-- 04
|   |   |-- 05
|   |   |-- 97
|   |   |-- 98
|   |   `-- 99
|   |-- 01
|   |   |-- 00
|   |   |-- 01

There is about 200 files at the end of each directory, so a lot of files. What would be the fastest way of making a tar.gz of the whole thing? Can I parallelize the task for many CPUs?

  • 6
    The bottleneck will be disk I/O, not CPU. Unless you have lots of small files on a RAID-1 device, you won't gain anything by parallelizing. Jun 28, 2011 at 12:40

4 Answers 4


You only need to specify one (or more) the top level directory(s) to include, and tar will recursively dip into all it's children and wrap everything up:

tar cvfz output.tar.gz /path/to/top_folder

(As @forcefsck notes if the object is raw speed you can leave off the v option for less verbosity on the terminal. By not echoing every file it adds you save a little bit if I/O time talking to your terminal.)

  • 4
    change cvfz to zcf. verbose output slows the process and f must be at the end so it is immediately followed by the archive filename.
    – forcefsck
    Jun 28, 2011 at 5:44
  • @forcefsck: You are right that the OP could leave off v, but unless the files are tiny it usually doesn't affect things that much and allows you to see that it did what you expected, which this OP needs to see. As for f, you are wrong, it does not need to be last. Try for yourself. Flags may come in any order as long as the next argument is the file name. In my head compression arguments like z and j makes a lot more sense that way.
    – Caleb
    Jun 28, 2011 at 6:14
  • about f, yes I am wrong :( somehow I remembered a case that it needed to be last. I guess my memory failed me. About v, the OP asked the fastest way and we do need to consider the case of many small files.
    – forcefsck
    Jun 28, 2011 at 7:24
  • @forcefsck: I edited my answer to not the speed consideration. Arugment order matters, the filename has to be the next argument, but arguments are separated by spaces. The wad of cvfz is a different syntax of flags. I think if you use the -c -v -z -f filename syntax you would have to do those in order. BSD's tar might only accept that syntax, I can't remember. However I have seen a lot of tutorials using my syntax but insisting f had to be last, which simply isn't the case.
    – Caleb
    Jun 28, 2011 at 7:55

You can pass the --fast, which is an alias to -1, option to gzip this way:

GZIP=--fast tar czf output.tar.gz /path/to/top_folder


GZIP=-1 tar czf output.tar.gz /path/to/top_folder

On some enviroments, the variable could be GZIP_OPT instead of GZIP.


On ext[234] filesystems, I find dump to be much faster than tar, especially with many small files, like a Maildir. It also uses multiple cores for concurrent compression. I suggest using -b 1024 for a 1mb block size, which works much better than the default 10kb block size.


Yes, using pigz:

 tar --use-compress-program="pigz --best --recursive" -cf file.tar.gz folder/
 tar -cf file.tar.gz -I pigz folder/

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .