6

An Oceanographer friend at work needs to back up many months worth of data. She is overwhelmed so I volunteered to do it. There are hundreds of directories to be backed up and we want to tar/bzip them into files with the same name as the directory. I can do this easy enough serially - but - I wanted to take advantage of the several hundred cores on my work station.

Question: using find with the -n -P args or GNU Parallel, how do I tar/bzip the directories, using as many cores as possible while naming the end product: origonalDirName.tar.bz2?

I have used find to bunzip 100 files simultaneously and it was VERY fast - so this is the way to approach the problem though I do not know how to get each filename to be that of each directory.

  • 2
    "several hundred cores on my work station" What? – Jodka Lemon Jul 20 '15 at 20:23
  • Just tar to stdout and pipe it to pigz. (You most likely don't want to parallelize disk access, just the compression part.) – PSkocik Jul 20 '15 at 20:30
  • @PSkocik pigz is an answer. Could you add a one liner, in an answer. – ctrl-alt-delor Jul 20 '15 at 20:34
  • 1
    You have never heard of multi core processors? And you don't know a robust work-station can have 30, 40 or more quad-core processors? And instead of looking it up - you say "What"? Good response Jodka. Thank you - your help is appreciated. – Peter Jul 21 '15 at 0:05
  • 4
    "30, 40, or more quad-core processors" is not a workstation. It is a rather expensive server, or a low-end mainframe. – Wyzard Jul 21 '15 at 3:42
6

Just tar to stdout and pipe it to pigz. (You most likely don't want to parallelize disk access, just the compression part.):

$ tar c- myDirectory/ | pigz > myDirectory.tar.gz

A plain tar invocation like the one above basically only concatenates directory trees in a reversible way. The compression part can be separate as it is in this example.

pigz does multithreaded compression. The number of threads it uses can be adjusted with -p and it'll default to the number of cores available.

1

pbzip2 works quite well. As with the answer above, tar to stdout and pipe to pbzip2:

$ tar -cf - mydir/ | pbzip2 > mydir.tar.bz2

pbzip2 accepts multiple options that allow for adjusting number of processors, amount of memory used, level of compression etc.

http://compression.ca/pbzip2/

Or for one archive per directory (assumes no spaces or special chars in directory names):

for dir in * ; do 
     [[ ! -d ${dir} ]] && continue
     tar cf -  ${dir} | bzip2 > ${dir}.tar.bz2 &
done
0

With GNU Parallel it looks like this:

parallel tar jcvf /tmp/{= s:/$:: =}.tar.bz2 {} ::: */

or:

parallel tar jcvf /tmp/{}.tar.bz2 {} ::: *

For better compression try:

parallel tar -I pxz -cvf /tmp/{= s:/$:: =}.tar.xz {} ::: */
  • Would you consider explaining the above commands? I am eager to learn/use GNU's Parallel - but the above (after the tar command) is not easily understood. I do thank you for your reply as well as all others. I have used the threaded versions of bzip2 and gzip - they were VERY fast. However - I had several files that were corrupted when I tried to extract. – Peter Jul 23 '15 at 1:33
  • Could you please explain or perhaps link to sources to understand what this regular expression means. Specifically : = s and /$. I understand ` *` checks for 0 or more spaces – GypsyCosmonaut Jul 27 '17 at 18:31
  • s:/$:: is a perl expression. It removes the ending / – Ole Tange Jul 27 '17 at 19:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.