0

I want to have an xargs-like interface to operate transparently on large tar balls without unpacking the whole archive at once. I've made already this shell script prototype xargs-tar which unpacks a tar ball into RAM-disk (/dev/shm), processes all files when they appear, and deletes all processed files immediately.

Here it is, xargs-tar:

#!/bin/bash

TAR_FILE="$1"
shift

TMP_ROOT="/dev/shm" # ...or /tmp
TMP_DIR="$(mktemp -d "$TMP_ROOT/xargs-tar-XXXXXX")"
UNTAR_DIR="$TMP_DIR/untar"
FILE_LIST="$TMP_DIR/files-list"
EXEC_FILE="$TMP_DIR/exec-file"

mkdir -p "$UNTAR_DIR"
mkfifo "$FILE_LIST"

(
    # single quotes for user's command and args
    for i in "$@"; do
            echo -n "'$i' "
    done
    echo '"$@"'
    echo 'rm "$@"'
) > "$EXEC_FILE"
chmod u+x "$EXEC_FILE"

# Background untar. Write file list (zero terminated, no directories) to named
# pipe. The output is one line delayed to make sure we print only finished file
# names.
(
    tar -v -C "$UNTAR_DIR" -xf "$TAR_FILE" \
            | awk 'BEGIN {last = "";}
                   !/\/$/ {if (last != "") print last; last=$0;}
                   END {if (last != "") print last;}' \
            | tr '\n' '\0'
) > "$FILE_LIST" &

cd "$UNTAR_DIR"
xargs --null -r sh "$EXEC_FILE" < "$FILE_LIST"

rm -rf "$TMP_DIR"

Example usage:

./xargs-tar  palemoon.tar.bz2  wc -l
      546 palemoon/libsoftokn3.so
     1437 palemoon/libnss3.so
[...]
      267 palemoon/libnssutil3.so
      220 palemoon/libsmime3.so
        6 palemoon/defaults/pref/channel-prefs.js
   379727 total

which should be equivalent (but faster and less disk-usage) to:

tar -xf palemoon.tar.bz2
find palemoon -type f -print0 | xargs -0 wc -l
rm -rf palemoon

Of course my xargs-tar prototype needs a lot of improvements, like

  • interface to limit max temp space (regarding available memory)
  • pausing untar when the consumer is too slow
  • error handling (duplicate files, whatever ...)
  • supporting other archive formats, not only tar
  • etc.

That's why I'm thinking about starting a serious project, implementation in C.

My question is now: Does something like this already exist? Would other people find it useful too? Do I waste my time?

I know tarfs, really useful but not exactly what I want. I want fast, simple piped command lines, portable implementation. The key is: The un-tarred files are processed while they are still cached, and then deleted immediately.

  • 2
    Let me ask you one question: Do you do those tars every minute or more often? If you do them few/several times per day I do not see any reason to put effort in such activity. At the end 99% of the computers are loaded 1 or 2% only – Romeo Ninov May 13 '17 at 4:29
  • I thought about this for a while, and I haven't found even a single use case for it. Counting files in an archive? Easy: tar tJvf palemoon.tar.bz2 | grep -v /$ | wc -l (or ... | grep ^- | ..., depending on what you want to count and how you want to deal with symlinks). All in all, RAM is much better left to the OS to use than filled up with temporary caches for tar files. – Satō Katsura May 13 '17 at 5:31
  • @SatoKatsura, they didn't count the files, but the lines in each file. – ilkkachu May 13 '17 at 10:35
  • @ilkkachu It's still a one-time operation. Each file is read at most once. Caching would start to make sense once each file was read at least twice. Caching in RAM would start to make sense once each file was read dozens of times. – Satō Katsura May 13 '17 at 14:17
  • 2
    Rather than “here's one way, are there others?”, I think you should ask “how do I do this?” and post your method as an answer. – Gilles May 13 '17 at 22:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.