2

I've got a bunch of .tar.gz files in different paths. I'd like to create a new .tar.gz file at some common ancestor of them and I don't want it to be composed of nested .tar.gz files. How can I easily flatten the archive once created?

  • 1
    (1) Extract everything, (2) Copy stuff you need to a new directory, (3) Create the archive. – devnull Feb 25 '14 at 2:57
2

Here is a bash script to recursively extract a tar archive, remove the original nested archives and create a new archive. It takes two arguments - first is the original archive, second is the name for the new archive. Both must be relative paths. This will extract the archive's directory, but will refuse to clobber any existing files (to do this remove the -k option from the tar command). Another approach to avoid clobbering would be to create a new directory for each archive and extract it there.

#!/bin/bash

archive="$1"
new_archive="$2"

# common extensions, full list at
# http://www.gnu.org/software/tar/manual/html_section/Compression.html#auto_002dcompress
match_archives='.*\.\(tar\|\(tar\.\(gz\|bz2\|xz\)\)\|\(tgz\|tbz\)\)$'

recursive_extract ()
{
  retval=0

  while read -rd '' path
  do
    if [ -e "$path" ]
    then
        nested_archive=${path##*/}
        if cd "${path%/*}" && tar -xakf "$nested_archive" 
        then
            rm "$nested_archive"
            find . -regex "$match_archives" -print0 | recursive_extract
            retval=$?
        else
            echo "Error extracting $nested_archive, not removing"
            retval=1
        fi
    fi
  done

  return $retval
}

tmpdir=$(mktemp -d) 
cd "$tmpdir"

tar -xaf "$OLDPWD/$archive" &&
  find . -regex "$match_archives" -print0 | recursive_extract &&
  tar -caf "$OLDPWD/$new_archive" * &&
  cd -- "$OLDPWD" &&
  rm -rf $tmpdir ||
  echo "Errors, please review $tmpdir"

Note if the extraction results in an error, it is possible for the above to attempt to extract the same archive multiple times.

1

You could do the following:

extract archive

$ mkdir tmpdir
$ tar xfz a.tar.gz -C tmpdir/

flatten to newdir

$ mkdir newdir
$ find tmpdir/ -type f -exec mv -i {} newdir/. \;

re-compress archive

$ tar zcvf somenew.tar.gz newdir/
0

With bsdtar and zsh:

set -o extendedglob # for (#i) case insensitive globbing operator
files=(**/*.(#i)(zip|7z|iso|cpio|a|ar|tar(|.[gx]z|.bz2)|t[bgx]z|tbz2)(D.))
bsdtar zcf result.tar.gz @$^files

(here including all the archive formats supported by bsdtar/libarchive).

Note that it doesn't recursively extract archives. If there's a a.tar.gz archive that contains a1.txt and a1.tar.gz and b.tar.gz with b1.txt and b1.tar.gz, result.tar.gz will contain a1.txt, a1.tar.gz, b1.txt, b1.tar.gz.

It will also happily include files with the same path several times.

Extracting the archives into a directory and re-archiving the resulting directory has a few issues:

  • if you're not root, you can't preserve the file ownerships and some attributes.
  • ownership is stored in tar archives as both the user name and user ids. Upon extracting, tar will by default try and use the user names and extract with the corresponding uid on the extracting system (and if there's no user by that name use the uid). So when recreating the archive, the uids may be different even if running as root.
  • Depending on the type of filesystem you're extracting the archive to, some file attributes like ACLs or extended attributes may be lost.
  • If the directory you're extracting the archives into has the sgid bit or default ACLs, those may end up affecting the files in the archive as well.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.