Merge possibly truncated gzipped log files

Question

I have multiple log files from each day that I need to merge together. Each comes from a different server. The job that puts them there sometimes gets interrupted and files get truncated. In that case the file gets written with a different name next time it runs. So I may end up with a list of log files like:

• server-1-log.gz (Yesterday's log file)
• server-1-log.1.gz (Today's log file that got interrupted while transferring and is truncated)
• server-1-log.2.gz (Today's log file re-transferred and intact)
• server-2-log.gz (Yesterday's log file)
• server-2-log.1.gz (Today's log file)

All the log files start with a time stamp on each line, so it is fairly trivial to sort and de-duplicate them. I've been trying to merge these files using the command:

zcat *.gz | sort | uniq | gzip > /tmp/merged.gz

The problem is that the truncated log file produces the following error from zcat:

gzip: server-1-log.1.gz: unexpected end of file

It turns out that zcat completely exits when it hits this error, without reading all the data from the other files. I end up losing the data that exists in the other good files because one of the files is corrupt. How can I fix this?

• Can I tell zcat not to exit on errors? I don't see anything in the man page for it.
• Can I fix truncated gzip files before calling zcat?
• Can I use a different decompression program instead?

Answer 1

I’m guessing you’re using the gzip script version of zcat. That just executes gzip -dc, which can’t be told to ignore errors and stops when it encounters one.

The documented fix for individual corrupted compressed files is to run them through zcat, so you won’t get much help there...

To process your files, you can either loop over them (with a for loop or xargs as you found), or use Zutils which has a version of zcat which continues processing when it encounters errors.

Answer 2

I found a way to do it. I can run each file through its own instance of zcat. To do so, I can use xargs -n 1 to start an instance of zcat for each file:

echo *.gz | xargs -n 1 zcat | sort | uniq | gzip > /tmp/merged.gz

The single zcat still fails, but the other ones run to completion. It doesn't kill the whole pipe.

Answer 3

Try Super Speedy Syslog Searcher

(assuming you have rust installed)

cargo install super_speedy_syslog_searcher

then

s4 server-*.gz

s4 can read .gz files.