21

I have a directory of logs that I would like to set up a job to compress using gzip. The issue is I don't want to recompress the logs I've already compressed.

I tried using ls | grep -v gz | gzip, but that doesn't seem to work.

Is there a way to do this? Basically I want to gzip every file in the directory that does not end in .gz.

3 Answers 3

29

You can just do:

gzip *

gzip will tell you it skips the files that already have a .gz ending.
If that message gets in the way you can use:

gzip -q *

What you tried did not work, because gzip doesn't read the filenames of the files to compress from stdin, for that to work you would have to use:

ls | grep -v gz | xargs gzip

You will exclude files with the pattern gz anywhere in the file name, not just at the end.¹ You also have to take note that parsing the output of ls is dangerous when you have file names with spaces, newlines, etc., are involved.

A more clean solution, not relying on gzip to skip files with a .gz ending is, that also handles non-compressed files in subdirectories:

find .  -type f ! -name "*.gz" -exec gzip {} \;



¹ As izkata commented: using .gz alone to improve this, would not work. You would need to use grep -vF .gz or grep -v '\.gz$'. That still leaves the danger of processing ls' output

3
  • 1
    "...that makes it less likely to match some internal part of a filename" - it's grep, just use $ so it's clearer. And you didn't escape the . anyway, so it's only going to exclude files that start with gz (since they don't have "any character before gz")
    – Izkata
    Commented Oct 10, 2014 at 21:35
  • 1
    @Izkata: No, including the unescaped dot and not anchoring the regex with $ going to exclude (-v) all files that have names consisting of three or more characters including any character followed by "gz" anywhere in the filename. So it will include files with names that start with "gz". Commented Oct 10, 2014 at 22:07
  • 1
    @DennisWilliamson My last comment wasn't clear, it's also going to exclude files with gz anywhere in the name (except the start). So it's better to anchor it anyway. Having not used $, it read as though you didn't realize what . means in a regex, hence the rest of the confusing comment.
    – Izkata
    Commented Oct 10, 2014 at 23:09
0

seems gzip does not have corresponding flag, but you could try this hack:

>gzip -9v * <&-
gzip: 20220710.txt.gz already exists;      not overwritten
20220710_old.txt:   84.2% -- replaced with 20220710_old.txt.gz
...

where

  • <&- suppresses input stream, which works like ultimate No answer
  • -9 max compression, could also use --best
  • v for verbose, to show whats going on with each file
3
  • <&- closes stdin (fd 0) which in general is not a wise thing to do, as that means that except on systems (libc, utilities) that guard against it, the first open()/socket()/pipe() will use that fd 0 (the first free one) and that will become stdin. For the ultimate "no answer", use < /dev/null instead. Commented Jun 14 at 9:29
  • @StéphaneChazelas - Thank you for comment, quite interesting and quite unexpectfull that fopen() would reassign stdin. Could you share some link to read more about such cases/hack please? I could not find it straight away(.
    – Fedor
    Commented Jun 17 at 19:51
  • See for instance nvd.nist.gov/vuln/detail/CVE-2024-27933, but vulnerabilities associated with closing fds 0, 1, 2 have been a thing since at least the 80s. The stories you'd hear about back then would be of setuid programs such as chsh that could end up opening /etc/shadow as fd 2 and would corrupt it when writing an error there. Commented Jun 18 at 5:44
0

Chances are you'd want to gzip only the files that are:

  • regular files, not directories or fifos or devices for instance. Not symlink either, even if they point to regular files, as compressing them would break symlinks. Some implementations/versions of gzip ignore or refuse to compress symlinks.
  • not already gzip-compressed, typically the ones with a .gz or .tgz extension, sometimes with variation of case if they've been through a MSDOS system for instance.
  • don't already have a corresponding .gz file including with variation of case for those file systems with case insensitive file names.
  • not hidden (hidden files often have a special meaning which they'd lose once compressed).
  • you may also want to skip files that are already compressed or for which using gzip would at best give you no gain such as jpeg, gif, png images, ogg, mp3 audio files, files compressed with other (often these days better) compression algorithms (xz, lzma, zstd, bz2...), or compressed archives (zip, jar, xlsx, rar, 7z...) though that may be beyond the scope of that Q&A.
  • you may also want to skip files whose size is less than that of a gzip header (files of size < 23 bytes are guaranteed to be larger once compressed by gzip).

In zsh, doing all those checks is relatively easy.

set -o extendedglob
gzip -- ^*.(#i)(gz|tgz)(.L+23^e['()(($#)) (#i)$REPLY.gz(NY1)'])

Where:

  • ^pattern negates a pattern, so here expand to the filenames that don't match the pattern (requires extendedglob)
  • (#i) turns on case-insensitivity for the rest of the pattern.
  • (a|b): alternation (a or b)
  • (NY1) and (.L...): glob qualifiers; not part of the pattern but lets you further select matching files based on other criteria than just their name:
    • N: nullglob: expands to nothing if there's no match rather than returning an error.
    • Y1: stop looking at the first hit (as an optimisation).
    • .: restrict to files of type regular
    • L+23: restrict to those whose Length is strictly greater than 23 bytes
    • ^...: negates the following qualifiers
    • e['code']: restrict to files for which the evaluation of code returns true (with the path of the file in $REPLY)
    • () {code} arguments: an anonymous function. Here the body of that function is the (($#)) arithmetic expression which returns true if $# (the number of arguments) is non-zero, so if the (#i)$REPLY.gz finds any match.

The -- guards against files whose name starts with - but not against the file called - which GNU gzip interprets as meaning stdin. Using gzip ./^*... would be better in that regard but would make the command line longer, increasing the likelihood that you run into a argument list too long error by the system.

As a limitation, note that while it won't compress file if File.gz already exists, it will happily try and compress both file and File.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .