I have multiple log files from each day that I need to merge together. Each comes from a different server. The job that puts them there sometimes gets interrupted and files get truncated. In that case the file gets written with a different name next time it runs. So I may end up with a list of log files like:
server-1-log.gz(Yesterday's log file)
server-1-log.1.gz(Today's log file that got interrupted while transferring and is truncated)
server-1-log.2.gz(Today's log file re-transferred and intact)
server-2-log.gz(Yesterday's log file)
server-2-log.1.gz(Today's log file)
All the log files start with a time stamp on each line, so it is fairly trivial to sort and de-duplicate them. I've been trying to merge these files using the command:
zcat *.gz | sort | uniq | gzip > /tmp/merged.gz
The problem is that the truncated log file produces the following error from
gzip: server-1-log.1.gz: unexpected end of file
It turns out that zcat completely exits when it hits this error, without reading all the data from the other files. I end up losing the data that exists in the other good files because one of the files is corrupt. How can I fix this?
- Can I tell
zcatnot to exit on errors? I don't see anything in the man page for it.
- Can I fix truncated gzip files before calling
- Can I use a different decompression program instead?