I have a massive number of files with extensions like .0_1234 .0_4213 and .0_4132 etc. Some of these are gzip compressed and some are raw email. I need to determine which are compressed files, decompress those, and rename all files to a common extension once all compress files are decompressed. I've found I can use the file command to determine which are compressed, then grep the results and use sed to whittle the output down to a list of files, but can't determine how to decompress the seemingly random extensions. Here's what I have so far

file *|grep gzip| sed -e 's/: .*$//g'

I'd like to use xargs or something to take the list of files provided in output and either rename them to .gz so they can be decompressed, or simply decompress them in-line.

  • Either move the files that are gzipped to a new/temp directory, or put the output of the list into a file. If you know the files are just text that is gzipped, then you could just loop through the list and zcat them out to a new file. Not sure how that would work with binary data though...
    – ivanivan
    Dec 22 '16 at 16:47

Don't use gzip, use zcat instead which doesn't expect an extension. You can do the whole thing in one go. Just try to zcat the file and, if that fails because it isn't compressed, cat it instead:

for f in *; do 
    ( zcat "$f" || cat "$f" ) > temp && 
    mv temp "$f".ext && 
    rm "$f" 

The script above will first try to zcat the file into temp and, if that fails (if the file isn't in gzip format), it will just cat it. This is run in a subshell to capture the output of whichever command runs and redirect it to a temp file (temp). Then, the temp is renamed to the original file name plus an extension (.ext in this example) and the original is deleted.

  • Also, you probably want an error check about "$f".ext already existing. Or use mv -i.
    – derobert
    Dec 22 '16 at 16:58
  • @don_crissti I know zcat uses gzip but on my system, at least, I can't get gzip to accept files without a .gz extension and this gets around that problem. How would I do it in one go?
    – terdon
    Dec 22 '16 at 17:00
  • @derobert true, although I assumed the OP wouldn't have taht since they want to rename automatically.
    – terdon
    Dec 22 '16 at 17:01
  • 1
    @don_crissti Argh! No I hadn't and I missed the essential detail that you were sending output to stdout. Yes indeed, that's a much better approach. Why don't you post it as an answer?
    – terdon
    Dec 22 '16 at 17:04
  • 2
    A difference between this and calling gunzip is that gunzip preserves the modification time. Dec 22 '16 at 23:55

You could do something like

for f in ./*
gzip -cdfq "$f" > "${f}".some_ext

This processes all files (even the uncompressed ones, via -f) and writes (via -c) the output to stdout using redirection to save the content of each file to its .some_ext counterpart. You could then remove the originals e.g. with bash

shopt extglob
rm -f ./!(*.some_ext)

or zsh

setopt extendedglob
rm -f ./^*some_ext

You could even save the resulting files to another dir (this time assuming you want to remove the original extension) e.g.

for f in *
gzip -cdfq -- "$f" > /some/place/else/"${f%.*}".some_ext

and then remove everything in the current dir...

  • Like Terdon's answer, this doesn't avoid copying the uncompressed files. But if you don't care about that, gzip's -f option to pass through non-gzipped data unchanged is the way to go, so +1. Dec 23 '16 at 3:20

This will present a list of all files compressed with gzip:

file /path/to/files | grep ': gzip compressed data' | cut -d: -f1

To tack a .gz extension on any gzipped files, this ugly hack might do:

for file in ./*; do
    if gzip -l "$file" > /dev/null 2>&1; then
        case "$file" in
          *.gz) :;; # The file already has the extension corresponding to its format
          *) mv "$file" "${file}.gz";;
        # Uncomment the following line to decompress them at the same time
        # gunzip "${file}.gz"

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.