Tell me more ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.
up vote 1 down vote favorite

I've got a tree of directories with files like this:

.
|-- 00
|   |-- 00
|   |   |-- 00
|   |   |-- 01
|   |   |-- 02
|   |   |-- 03
|   |   |-- 04
|   |   |-- 05
[..]
|   |   |-- 97
|   |   |-- 98
|   |   `-- 99
|   |-- 01
|   |   |-- 00
|   |   |-- 01
[...]

There is about 200 files at the end of each directory, so a lot of files. What would be the fastest way of making a tar.gz of the whole thing? Can I parallelize the task for many CPUs?

share|improve this question
5  
The bottleneck will be disk I/O, not CPU. Unless you have lots of small files on a RAID-1 device, you won't gain anything by parallelizing. – Gilles Jun 28 '11 at 12:40

2 Answers

up vote 2 down vote accepted

On ext[234] filesystems, I find dump to be much faster than tar, especially with many small files, like a Maildir. It also uses multiple cores for concurrent compression. I suggest using -b 1024 for a 1mb block size, which works much better than the default 10kb block size.

share|improve this answer
up vote 5 down vote

You only need to specify one (or more) the top level directory(s) to include, and tar will recursively dip into all it's children and wrap everything up:

tar cvfz output.tar.gz /path/to/top_folder

(As @forcefsck notes if the object is raw speed you can leave off the v option for less verbosity on the terminal. By not echoing every file it adds you save a little bit if I/O time talking to your terminal.)

share|improve this answer
3  
change cvfz to zcf. verbose output slows the process and f must be at the end so it is immediately followed by the archive filename. – forcefsck Jun 28 '11 at 5:44
@forcefsck: You are right that the OP could leave off v, but unless the files are tiny it usually doesn't affect things that much and allows you to see that it did what you expected, which this OP needs to see. As for f, you are wrong, it does not need to be last. Try for yourself. Flags may come in any order as long as the next argument is the file name. In my head compression arguments like z and j makes a lot more sense that way. – Caleb Jun 28 '11 at 6:14
about f, yes I am wrong :( somehow I remembered a case that it needed to be last. I guess my memory failed me. About v, the OP asked the fastest way and we do need to consider the case of many small files. – forcefsck Jun 28 '11 at 7:24
@forcefsck: I edited my answer to not the speed consideration. Arugment order matters, the filename has to be the next argument, but arguments are separated by spaces. The wad of cvfz is a different syntax of flags. I think if you use the -c -v -z -f filename syntax you would have to do those in order. BSD's tar might only accept that syntax, I can't remember. However I have seen a lot of tutorials using my syntax but insisting f had to be last, which simply isn't the case. – Caleb Jun 28 '11 at 7:55

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.