How can I filter the contents of a tar file, producing another tar file in the pipe?

Question

Consider a single tar file from an external system which contains some directories with various attributes which I want to retain such as permissions, mtimes, etc. How can I easily take a subset of these files as a regular user (not root)?

Looking for something like:

tar -f some.tar.gz --subset subdir/ | ssh remote@system tar xvz

It is also essential that the main attributes (ownership, group, mode, mtime) in this tar archive are retained. What about other attributes in a tar file such as extended header keywords?

Bonus points for a solution that avoids use of a temporary directory in case this subdir contains huge files.

Answer 1

bsdtar (based on libarchive) can filter tar (and some other archives) from stdin to stdout. It can for example pass through only filenames matching a pattern, and can do s/old/new/ renaming. It's already packaged for most distros, for example as bsdtar libarchive-tools in Ubuntu.

sudo apt-get install libarchive-tools   # or aptitude, if you have it.

# example from the man page:
bsdtar -c -f new.tar --include='*foo*' @old.tgz
#create new.tar containing only entries from old.tgz containing the string ‘foo’
bsdtar -czf - --include='*foo*' @-  # filter stdin to stdout, with gzip compression of output.

Note that has a wide choice of compression formats for input/output, so you don't have to manually pipe through gunzip / lz4 yourself. You can use - for stdin with the @tarfile syntax, and/or - for stdout like normal.

---

My searching also found this streaming tar modify tool which appears to want you to define the archive changes you want using javascript. (I think the whole thing is written in js).

https://github.com/mafintosh/tar-stream

Answer 2

The easiest way would be to copy the whole archive; I presume you don't want to do that because it's too large.

The usual command line tools (tar, pax) don't support copying members of an archive to another archive.

If you didn't need to preserve ownership, I'd suggest using FUSE filesystems. You can use archivemount to mount an archive as a filesystem; do this for the source archive, and run tar on the mounted filesystem.

archivemount some.tar.gz mnt
cd mnt
tar -cz subdir | ssh example.com tar -xz
fusermount -u mnt

Alternatively, you can use AVFS:

mountavfs
cd ~/.avfs$PWD/some.tar.gz\#
tar -cz subdir | ssh example.com tar -xz

Alternatively, you can run tar on the original archive and extract to the remote machine over SSHFS.

sshfs example.com: mnt
cd mnt
tar -xf /path/to/some.tar.gz subdir
fusermount -u mnt

However all of these methods are cumbersome if you need to preserve ownership. They all involve extracting to a file on the local machine, so this file's ownership will have to be the intended remote ownership. This requires running as root and may not give the intended result if the files are owned by accounts which have names or IDs that differ between the local machine and the remote host.

Python's tarfile library provides a fairly easy way to manipulate tar members, so you can shuffle them from one tar file to another. It supports POSIX standard formats (ustar, pax) as well as some GNU extensions. Here's an untested Python script that reads a tar file (possibly compressed with gzip or bzip2) on its standard input and writes a tar file compressed with bzip2 on its standard output. The members from the source are copied if they start with the argument passed to the script.

#!/usr/bin/env python2
import sys, tarfile
source = tarfile.open(fileobj=sys.stdin)
destination = tarfile.open(fileobj=sys.stdout, mode='w:bz2')
for info in source:
    if info.name.startswith(sys.argv[1]):
        destination.addfile(info)
destination.close()

To be invoked as

tar_filter <some.tar.gz subdir/ | ssh example.com tar -xj

Answer 3

GNU tar does have a --delete option:

$ tar -c a b c | tar --delete a | tar -t
b
c

This way, you can get a subset of the input tar by specifying what not to include in the output.

Unfortunately I couldn't get the --exclude option to work with --delete, so it seems you first need to get an explicit list (-t) of things to delete and then pass it to another invocation of tar.

$ tar --delete --no-recursion `tar -t --exclude subdir <some.tar` <some.tar | ssh ...

Or you can store the list in an external file if it is too long or complex:

$ tar -t --exclude subdir <some.tar >to_delete.lst
$ tar --delete --no-recursion -T to_delete.lst <some.tar | ssh ...

Answer 4

An alternative privilege-less approach is to use the fakeroot program to pretend that you are allowed to change ownership. While other tar attributes are lost, it does keep the mode, mtime and uid/gid. These commands create a temporary directory, extract a subset of the files and finally creates a new archive:

mkdir tmp
<some.tar.gz \
fakeroot -- sh -c 'cd tmp && tar -xzf- subdir/ && tar -czf- subdir' |
   ssh remote@system tar -xzvf-
rm -rf tmp

Answer 5

From what I know, the tar command cannot use the tar format both as input and output. You will have to extract your files locally somehow, and use tar again to create a tarfile on-the-fly, with something like this (the - means standart input/output is used instead of a file) :

tar cf - subdir/ | ssh remote@system 'cd extractdir && tar xvf -'

Note that having tar being able to extract a tarfile directly in another tarfile is an interesting idea...