2

I'm trying to compress a large rootdir which contains many subdir_i the folder tree looks like:

./rootdir
./rootdir/subdir_1
./rootdir/subdir_2
...

I am looking to output this to a single compressed archive but having each subdirectory in it's own tar archive:

rootdir.tar.xz  # containing:
     subdir_1.tar
     subdir_2.tar

I've tried the following :

for foo in `find rootdir -maxdepth 1 -name "subdir_*" -type d`
do
     tar --create --verbose --file=- --directory="rootdir" `basename ${foo}` 
     # in shorter form:  tar -cvf - -C rootdir subdir_i
done | xz -zc9 > rootdir.tar.xz 

Which does isolate the subdirectory into an xz but a single tar archive is inside with only the last directory:

rootdir.tar.xz
     rootdir.tar # containing subdir_2/

However the size of that archive is consistent with compression of the whole rootdir tree. Any ideas why that is and how to get it to do what I want (without using intermediate archives)?

3
  • I don't think xz files can contain more than one file, so all you get after uncompressing it is a bunch of concatenated tar files, and I'm not sure that's any better than just one tar file? What are you trying to do here? Have an easy way to extract any individual subdirectory from the final archive?
    – ilkkachu
    May 12, 2017 at 15:21
  • Exactly, be able to uncompress only part of the data.
    – adjpayot
    May 12, 2017 at 16:06
  • @adjpayot even using only one tar you are still able to extract only part of the data. But tar is not performing best in this case. I would simply keep all the subdir tar balls separated.
    – rudimeier
    May 12, 2017 at 16:16

2 Answers 2

2

You can create tar files for each directory, i.e., directory-1.tar, directory-2.tar, ..., , directory-n.tar, then create a tar file with all the pieces and compress it by tar Jcf whole-shebang.tar.xz directory-*.tar.

Why do this, when you can just tar Jcf whole-shebang.tar.xz directory-*? You can certainly extract individual directories/files from that hairball, in your setup you'd have to first extract the specific tarball and then rummage in it to get what you want.

1

What your are actually doing is to concatenate all the tar balls. You can see the whole content using tar's -i switch:

$ tar -itvf rootdir.tar.xz
subdir_2/
subdir_1/

I don't think it's possible to create a tar ball of all these subdir tar balls directly via a pipe. You would need to write them to disk first within the loop:

for foo in `find rootdir -maxdepth 1 -name "subdir_*" -type d`
do
  tar --create --verbose --file=- --directory="rootdir" "$(basename ${foo})" \
  >"${foo}.tar"
done
tar -cf -  rootdir/subdir*.tar | xz -zc9 > rootdir.tar.xz
rm rootdir/subdir*.tar
1
  • It does seem to be all there, for some reason it isn't showing up in 7z. I need to do that simultaneously to up to 2TB of data. Making a copy is not an option.
    – adjpayot
    May 12, 2017 at 16:05

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .