I'm trying to compress a large rootdir which contains many subdir_i the folder tree looks like:


I am looking to output this to a single compressed archive but having each subdirectory in it's own tar archive:

rootdir.tar.xz  # containing:

I've tried the following :

for foo in `find rootdir -maxdepth 1 -name "subdir_*" -type d`
     tar --create --verbose --file=- --directory="rootdir" `basename ${foo}` 
     # in shorter form:  tar -cvf - -C rootdir subdir_i
done | xz -zc9 > rootdir.tar.xz 

Which does isolate the subdirectory into an xz but a single tar archive is inside with only the last directory:

     rootdir.tar # containing subdir_2/

However the size of that archive is consistent with compression of the whole rootdir tree. Any ideas why that is and how to get it to do what I want (without using intermediate archives)?

  • I don't think xz files can contain more than one file, so all you get after uncompressing it is a bunch of concatenated tar files, and I'm not sure that's any better than just one tar file? What are you trying to do here? Have an easy way to extract any individual subdirectory from the final archive? – ilkkachu May 12 '17 at 15:21
  • Exactly, be able to uncompress only part of the data. – adjpayot May 12 '17 at 16:06
  • @adjpayot even using only one tar you are still able to extract only part of the data. But tar is not performing best in this case. I would simply keep all the subdir tar balls separated. – rudimeier May 12 '17 at 16:16

What your are actually doing is to concatenate all the tar balls. You can see the whole content using tar's -i switch:

$ tar -itvf rootdir.tar.xz

I don't think it's possible to create a tar ball of all these subdir tar balls directly via a pipe. You would need to write them to disk first within the loop:

for foo in `find rootdir -maxdepth 1 -name "subdir_*" -type d`
  tar --create --verbose --file=- --directory="rootdir" "$(basename ${foo})" \
tar -cf -  rootdir/subdir*.tar | xz -zc9 > rootdir.tar.xz
rm rootdir/subdir*.tar
  • It does seem to be all there, for some reason it isn't showing up in 7z. I need to do that simultaneously to up to 2TB of data. Making a copy is not an option. – adjpayot May 12 '17 at 16:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.