4

I have 6 gzipped text files, each of which is ~17G when compressed. I need to see the last few lines (decompressed) of each file to check whether a particular problem is there. The obvious approach is very slow:

for i in *; do zcat "$i" | tail -n3; done

I was thinking I could do something clever like:

for i in *; do tail -n 30 "$i" | gunzip | tail -n 4 ; done

Or

for i in *; do tac "$i" | head -100 | gunzip | tac | tail -n3; done

But both complain about:

gzip: stdin: not in gzip format

I thought that was because I was missing the gzip header, but this also fails:

$ aa=$(head -c 300 file.gz)
$ bb=$(tail -c 300 file.gz)
$ printf '%s%s' "$aa" "$bb" | gunzip
gzip: stdin: unexpected end of file

What I am really looking for is a ztail or ztac but I don't think those exist. Can anyone come up with a clever trick that lets me decompress and print the last few lines of a compressed file without decompressing the entire thing?

  • 1
    I don't think you can... See also How can I tail a zipped file without reading its entire contents? and the links there... – don_crissti Jun 28 '16 at 10:23
  • @don_crissti well, dang! Looks like you're right. I'll leave this for a while in case anyone comes up with something really clever, and if not I'll copy one of the SO answers over. Probably the one stating that gzip doesn't support this. Feel free to do it yourself, by the way. – terdon Jun 28 '16 at 10:27
6

You can't, as it has been already said, if the files have been compressed with standard gzip. If you have control over the compression, you can use dictzip to compress the files, it compresses the files in separate blocks and you can decompress just the last block (typically 64KB). And it is backward compatible with gzip, meaning the dictzipped file is perfectly legal gzipped file as well.

Other possibility would be if you get the gzipped file as a concatenation of several already gzipped files, you could search for the last gzip signature and decompress everything after that.

  • I don't, sadly. I was hoping I could use the idea here to get the right chunks but I don't know if that's possible or what the chunks might be. – terdon Jun 28 '16 at 15:05
  • @terdon The idea there is to create the compressed file in chunks and then only uncompress the last chunk. If the compressed file has a single chunk, which is the case if you produced it with gzip, then “only uncompress the last chunk” = “uncompress the sole chunk” = “uncompress the whole file”. – Gilles Jun 28 '16 at 21:29
  • @Gilles I see. I had understood that the "chunks" were a result of the gzip algorithm and could perhaps be inferred from the compressed file. – terdon Jun 29 '16 at 8:54
2

Well, you can access randomly a gzipped file if you previously creates an index for each file ...

I've developed the command line tool that you were probably looking for: it can access the tail using the same amount of time than a gunzip... BUT because it creates a little (<<1%/gzip) index, next extractions will be really quick:

https://github.com/circulosmeos/gztool

The tool has two options that may be of interest for you:

  • -S option supervise a still-growing file and creates an index for it as it is growing - this can be useful for gzipped rsyslog files as reduces to zero in the practice the time of index creation.
  • -t tails a gzip file: this way you can do: $ gztool -t foo.gz Please, note that if the index doesn't exists, this will consume the same time as a complete decompression: but as the index is reusable, next searches will be greatly reduced in time - and as it is the same time, why not use it and create the index at the same time?

This tool is based on zran.c demonstration code from original zlib, so there's no out-of-the-rules magic!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.