I have multiple log files from each day that I need to merge together. Each comes from a different server. The job that puts them there sometimes gets interrupted and files get truncated. In that case the file gets written with a different name next time it runs. So I may end up with a list of log files like:

  • server-1-log.gz (Yesterday's log file)
  • server-1-log.1.gz (Today's log file that got interrupted while transferring and is truncated)
  • server-1-log.2.gz (Today's log file re-transferred and intact)
  • server-2-log.gz (Yesterday's log file)
  • server-2-log.1.gz (Today's log file)

All the log files start with a time stamp on each line, so it is fairly trivial to sort and de-duplicate them. I've been trying to merge these files using the command:

zcat *.gz | sort | uniq | gzip > /tmp/merged.gz

The problem is that the truncated log file produces the following error from zcat:

gzip: server-1-log.1.gz: unexpected end of file

It turns out that zcat completely exits when it hits this error, without reading all the data from the other files. I end up losing the data that exists in the other good files because one of the files is corrupt. How can I fix this?

  • Can I tell zcat not to exit on errors? I don't see anything in the man page for it.
  • Can I fix truncated gzip files before calling zcat?
  • Can I use a different decompression program instead?

3 Answers 3


I’m guessing you’re using the gzip script version of zcat. That just executes gzip -dc, which can’t be told to ignore errors and stops when it encounters one.

The documented fix for individual corrupted compressed files is to run them through zcat, so you won’t get much help there...

To process your files, you can either loop over them (with a for loop or xargs as you found), or use Zutils which has a version of zcat which continues processing when it encounters errors.

  • Is the "continue processing on errors" from zutils standard, or does it need a command line argument for it? Oct 20, 2017 at 16:56
  • It’s the default behaviour, at least for zcat. Oct 20, 2017 at 17:08
  • Thanks. zutils just got installed on all my servers. Seems like a nice package with significant functionality over the standard stuff. Oct 20, 2017 at 17:13

I found a way to do it. I can run each file through its own instance of zcat. To do so, I can use xargs -n 1 to start an instance of zcat for each file:

echo *.gz | xargs -n 1 zcat | sort | uniq | gzip > /tmp/merged.gz

The single zcat still fails, but the other ones run to completion. It doesn't kill the whole pipe.


Try Super Speedy Syslog Searcher

(assuming you have rust installed)

cargo install super_speedy_syslog_searcher


s4 server-*.gz

s4 can read .gz files.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .