67

When handling log files, some end up as gzipped files thanks to logrotate and others not. So when you try something like this:

$ zcat *

you end up with a command line like zcat xyz.log xyz.log.1 xyz.log.2.gz xyz.log.3.gz and then with:

gzip: xyz.log: not in gzip format

Is there a tool that will take the magic bytes, similar to how file works, and use zcat or cat depending on the outcome so that I can pipe the output to grep for example?

NB: I know I can script it, but I am asking whether there is a tool out there already.

10 Answers 10

39

zless

It seems a pity about zcat, as libz has an API that supports reading from both compressed and uncompressed files transparently. But the manpage does say that zcat is equivalent to gunzip -c.

  • Thank you for this alternative. I could have thought of that, could I not? ;) ... oh well. Spot on, +1 and accept (also because you have less rep than the other answerer). – 0xC0000022L May 27 '13 at 20:10
  • Amazing. Helped me a lot, I was using shell script to solve that for years... or a terrible perl script... log resolve merge, used by awstats... now i know this amazing tool. Thanks. – Luciano Andress Martini Sep 3 '18 at 20:13
93

Try it with -f or --force:

zcat -f -- *

Since zcat is just a simple script that runs

exec gzip -cd "$@"

with long options that would translate to

exec gzip --stdout --decompress "$@"

and, as per the man gzip (emphasize mine):

-f --force
      Force compression or decompression even if the file has multiple links
      or the corresponding file already exists, or if the compressed data is
      read from or written to a terminal. If the input data is not in a format
      recognized by gzip, and if the option --stdout is also given, copy the
      input data without change to the standard output: let zcat behave as cat.

Also:

so that I can pipe the output to grep for example

You could use zgrep for that:

zgrep -- PATTERN *

though see Stéphane's comment below.

  • 1
    Thanks, that's an interesting alternative to the zless solution. Nice and +1. – 0xC0000022L May 27 '13 at 20:08
  • 6
    Note that both zless and zgrep are scripts that do call gzip -cdfq (that is zcat -fq). – Stéphane Chazelas May 28 '13 at 12:50
7

I use exactly for the same purpose:

{ cat /var/log/messages ; zcat /var/log/messages*.gz ; }| grep something | grep "something else" ....
  • I like this approach because, it requires the least time spent on educating co-workers. If the log messages have a timestamp in a sort-friendly timestamp, this is especially useful. – Thomas L Holaday Feb 26 '18 at 18:18
6

There is a drop-in replacement for ztools (zcat, zgrep, ..) called zutils that unites all the decompression tools independently of the backend. So with the same command you can read plain, lzma, gzipped, xz files transparently.

It's available in debian wheezy or newer, probably in redhat/centos too.

The project's page is here nongnu.org

A blog post explaining the usage of the util here (noone.org)

3

This works fine in RHEL 5.x where zcat is a binary. It fails in RHEL 6.x (and Ubuntu 12.x) where zcat is a script. This used to work fine.

I wouldn't be using zcat at all but zgrep won't properly handle uncompressed files either.

1

What about wrapper?

$ cat xcat.sh 
#!/bin/bash

for i in $@;do 
        [ ! -z "$(file -i $i | grep "gzip")" ] && zcat $i || cat $i
done

$ bash xcat.sh plain.txt gzipped_text.gz
1

Opens both compressed and non-compressed, in chronological order.

ls -v syslog* | tac | xargs zcat -f | less
  • It gives wrong order with more than ten log files (syslog.10.gz ...) – Vanni Aug 19 '15 at 22:51
  • Good catch. -v should fix that. – Ryan Sep 1 '15 at 3:44
  • ls -rv to avoid tac. For log files, less $(ls -rv syslog*) with your LESSOPEN env var set properly works well. You can do search across files with esc-n to find next match, ignoring file boundaries. – Peter Cordes Sep 1 '15 at 4:46
  • With zsh: zcat -f syslog*(nOn) – Stéphane Chazelas May 10 '17 at 16:33
  • This doesn't work if you have your log rotate set to compress the next day – cjbarth Sep 20 '17 at 16:22
0

Copy and paste (or put it at the end of your ~/.bashrc file) this bash function:

logs() { zcat -f $(ls -rv "$1"*) | less; }

Now you can type for example logs /var/log/syslog or logs /var/log/nginx/access.log to see all the syslog or nginx log messages from oldest to newest with less.

You can then search something typing /something and hitting n for next.

0

There is a beautiful perl script exacly doing this. It's logresolvemerge.pl from the awstats project: http://www.awstats.org/docs/awstats_tools.html

Logresolvemerge allows you to get one unique output log file, sorted on date, built from particular sources:

  • It can read several input log files
  • It can read .gz/.bz2 log files

    The output is on STDOUT, so that you can utilize it quite nicely in additional processes.

  • We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

    0

    Building on @Ryan's answer, the following will get the get all the 'rolled' files sorted alphabetically, then get the current file, uncompress them, if needed, and less them:

    cat <(ls mylog.log-* | sort) <(ls mylog.log) | xargs zcat -f | less

    or if you want to get them all as a continuous stream, you can tail them, and optionally pipe that to another process

    cat <(ls mylog.log-* | sort | xargs zcat -f) <(tail -f -n +0 mylog.log)

    I should note that this is designed for logs that are rotated daily with the date appended to the end of the file. If your logs us a different format, you'll have to modify the first part of the cat statement to accommodate.

    Your Answer

    By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

    Not the answer you're looking for? Browse other questions tagged or ask your own question.