mass decompress gzip files without gz extension

Question

I have a massive number of files with extensions like .0_1234 .0_4213 and .0_4132 etc. Some of these are gzip compressed and some are raw email. I need to determine which are compressed files, decompress those, and rename all files to a common extension once all compress files are decompressed. I've found I can use the file command to determine which are compressed, then grep the results and use sed to whittle the output down to a list of files, but can't determine how to decompress the seemingly random extensions. Here's what I have so far

file *|grep gzip| sed -e 's/: .*$//g'

I'd like to use xargs or something to take the list of files provided in output and either rename them to .gz so they can be decompressed, or simply decompress them in-line.

Answer 1

Don't use gzip, use zcat instead which doesn't expect an extension. You can do the whole thing in one go. Just try to zcat the file and, if that fails because it isn't compressed, cat it instead:

for f in *; do 
    ( zcat "$f" || cat "$f" ) > temp && 
    mv temp "$f".ext && 
    rm "$f" 
done

The script above will first try to zcat the file into temp and, if that fails (if the file isn't in gzip format), it will just cat it. This is run in a subshell to capture the output of whichever command runs and redirect it to a temp file (temp). Then, the temp is renamed to the original file name plus an extension (.ext in this example) and the original is deleted.

Answer 2

You could do something like

for f in ./*
do 
gzip -cdfq "$f" > "${f}".some_ext
done

This processes all files (even the uncompressed ones, via -f) and writes (via -c) the output to stdout using redirection to save the content of each file to its .some_ext counterpart. You could then remove the originals e.g. with bash

shopt extglob
rm -f ./!(*.some_ext)

or zsh

setopt extendedglob
rm -f ./^*some_ext

You could even save the resulting files to another dir (this time assuming you want to remove the original extension) e.g.

for f in *
do 
gzip -cdfq -- "$f" > /some/place/else/"${f%.*}".some_ext
done

and then remove everything in the current dir...

Answer 3

This will present a list of all files compressed with gzip:

file /path/to/files | grep ': gzip compressed data' | cut -d: -f1

To tack a .gz extension on any gzipped files, this ugly hack might do:

for file in ./*; do
    if gzip -l "$file" > /dev/null 2>&1; then
        case "$file" in
          *.gz) :;; # The file already has the extension corresponding to its format
          *) mv "$file" "${file}.gz";;
        esac
        # Uncomment the following line to decompress them at the same time
        # gunzip "${file}.gz"
    fi
done