This question already has an answer here:

I have a directory with several hundred sub-directories. I want to tar and compress each sub-directory and name the resulting file <currentDirName>.tar.bz2.

I have done:

find ./dir -type f -print0 | xargs -0 -n1 -P100 bunzip2

to decompress 100 files at one time. (I do ocean modeling so my machine is quite powerful with many quad cores.) I do not know how to do something like the above while including information on naming the compressed archive file on the fly. I don't want to do tar cfj dir.tar.bz2 dir for each one.

Can I do it with each directory name being used automatically as the tar file name - via either find or parallel so that 100 or more can be done at a time as in the bunzip command above?

Thank you for input.


marked as duplicate by dhag, Gilles, roaima, slm Jul 21 '15 at 23:13

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 1
    The tar process will mostly wait for disk, but when invoked with the -j option it will spawn bzip2 which probably will consume 100% of 1 core. It is possible to split the input for bzip2 and concatenate outputs of bzip2 and that can be used to parallelize compression with tar. Instead of calling tar with -j you can pipe its output to bzip2 or gnu parallell calling bzip2. – Henrik Carlqvist Jul 21 '15 at 17:18

The following bash one-liner will do approximately what you describe, putting each directory into its own tarball.

for d in dir/*/; do { tar -cj "$d" > "${d%/}.tar.bz2" ; } & done ; while [ "$(jobs)" ] ; do fg &>/dev/null ; done ; echo done

Not the answer you're looking for? Browse other questions tagged or ask your own question.