3

I am sending compressed file with piping either local or from a network location. And on the receiving end, I would like to detect the type of compression and use the appropriate decompression utility (gzip, bzip2, xz..etc) to extract it. Commands looks as follows:

Local:

cat misteryCompressedFile | [compressionUtility] -d -fc > /opt/files/uncompressedfile

Over network:

ssh user@ipaddr "cat misteryCompressedFile" | [compressionUtility] -d -fc > /opt/files/uncompressedfile

One can tell the type of compression used even if there is no extension provided (e.g., .gz or .bz2) by looking at first few hex values of the file. For example, if I use xxd to look at first few hex values of two compressed files, then I will 1f8b 0808 for gzip and 425a 6836 for bzip2.

However, to still use piping, how can I check the first incoming byte to select the proper decompression utility for the first of the file?

So if unknown compressed file is a gzip type, command will be this:

cat misteryCompressedFile | gzip -d -fc > /opt/files/uncompressedfile

and if unknown compressed file is bzip2 type, command will be this:

cat misteryCompressedFile | bzip2 -d -fc > /opt/files/uncompressedfile

Is it possible to make such decision with piping on the fly without having to download entire file and then make decision what to use for decompression?

4

Yes, you can do that in the pipeline, without having to read the whole file.

This first script fragment illustrates the mechanism by which we will intercept and inspect the header and pass it on. Notice that we print the header to stderr (>&2), yet it continues to appear in the output:

$ echo 0123456789ABCDEF |
(
    HEADER=$(dd bs=1 count=4);
    printf 'HEADER:%s\n' "$HEADER" >&2;
    printf '%s\n' "$HEADER";
    cat 
)
4+0 records in
4+0 records out
4 bytes (4 B) copied, 8.4293e-05 s, 47.5 kB/s
HEADER:0123
0123456789ABCDEF
$

The key is using the dd the file conversion utility with a small block size bs=1.

Expanding on that, this is a working solution. We'll use a temporary file to store the binary header. If it doesn't see one of the two 4-byte headers then it does nothing:

#!/bin/sh

trap "rm -f /tmp/$$; exit 1" 1 2 3 15

# grab the 1st 4 bytes off the input stream,
# store them in a file, convert to ascii,
# and store in variable:
HEADER=$(
    dd bs=1 count=4 2>/dev/null |
    tee /tmp/$$ |
    od -t x1 |
    sed '
        s/^00* //
        s/ //g
        q
    '
)

case "$HEADER" in
    1f8b0800)
        UNCOMPRESS='gzip -d -fc'
    ;;
    425a6839)
        UNCOMPRESS='bzip2 -d -fc'
    ;;
    *)
        echo >&2 "$0: unknown stream type for header '$HEADER'"
        exit 2
    ;;
esac

echo >&2 "$0: File header is '$HEADER' using '$UNCOMPRESS' on stream."
cat /tmp/$$ - | $UNCOMPRESS
rm /tmp/$$
  • nice. still have to somehow use that info to decide which decompression tool to use. – cas Nov 3 '15 at 3:19
  • OK. Let me go make some compressed files and put my binary hat on! :-) – RobertL Nov 3 '15 at 3:26
  • Note that this will mangle the header if it contains null bytes, because shell variables can't contain null bytes (except in zsh). – Gilles 'SO- stop being evil' Nov 3 '15 at 21:12
  • @Gilles No, because the file header passed to the selected uncompress command is stored in a temporary file. The shell variable is used only to select the uncompress command. – RobertL Nov 3 '15 at 21:15
  • Indeed your second script is fine, I was refering to the first one, sorry I wasn't precise enough. – Gilles 'SO- stop being evil' Nov 3 '15 at 21:17
1

Use file on the sending machine and use that info to decide which decompression command to run on the remote host.

e.g.

#! /bin/sh

filetype=$(file misteryCompressedFile)

case "$filetype" in
   *gzip*) CMD='gzip' ; ARGS='-d -fc' ;;
   *bzip2*) CMD='bzip2' ; ARGS='-d -fc' ;;
   *) echo "error: unknown compression type" ; exit 1 ;;
esac

cat misteryCompressedFile | ssh user@ipaddr "$CMD $ARGS > /opt/files/uncompressedfile"

In the example shown, the ARGS for gzip and bzip2 commands are the same...but they may be different for other decompression tools.

Here's a version that will decompress a file fetched from the remote host:

#! /bin/sh

# set up an anonymous fifo on fd 3 so we can pass the 
# output of `file` to the second subshell without risking
# corruption of stdout/stdin
FIFO=$(mktemp -u)
mkfifo "$FIFO"
exec 3<>"$FIFO"
rm -f "$FIFO"

ssh user@ipaddr 'cat misteryCompressedFile' | 
(
    HEADER=$(dd bs=1 count=20 2> /dev/null | 
             od -A none -t o1 -w512 | 
             sed -e 's: :\\:g')

    printf "$HEADER" | file --mimetype - | cut -d/ -f2 >&3
    printf "$HEADER"
    cat
) | (
    read -u 3 -r filetype
    case "$filetype" in
       gzip) CMD='gzip' ; ARGS='-d -fc' ;;
       x-bzip2) CMD='bzip2' ; ARGS='-d -fc' ;;
       x-xz) CMD='unxz' ; ARGS='' ;;
       x-lzma) CMD='lzcat' ; ARGS='' ;;
       x-compress) CMD='uncompress' ; ARGS='' ;;
       x-lrzip) CMD='lrzcat' ; ARGS='' ;;
       *) echo "error: unknown compression type" >&2 ; exit 1 ;;
    esac

    $CMD $ARGS > /opt/files/uncompressedfile
)
  • Thanks. But I am making request on the client (to where file is downloaded). On remote host, there is only cat utility or other utility/script for sends file into stdout. So in away client has no idea what file is piped to it.... – Armen Nov 3 '15 at 3:27
  • @Armen, I've added another version of the script that does what you want on the local machine. it makes use of RobertL's idea of using dd in a subshell to extract the files header. – cas Nov 3 '15 at 4:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.