3

I have 6 gzipped text files, each of which is ~17G when compressed. I need to see the last few lines (decompressed) of each file to check whether a particular problem is there. The obvious approach is very slow:

for i in *; do zcat "$i" | tail -n3; done

I was thinking I could do something clever like:

for i in *; do tail -n 30 "$i" | gunzip | tail -n 4 ; done

Or

for i in *; do tac "$i" | head -100 | gunzip | tac | tail -n3; done

But both complain about:

gzip: stdin: not in gzip format

I thought that was because I was missing the gzip header, but this also fails:

$ aa=$(head -c 300 file.gz)
$ bb=$(tail -c 300 file.gz)
$ printf '%s%s' "$aa" "$bb" | gunzip
gzip: stdin: unexpected end of file

What I am really looking for is a ztail or ztac but I don't think those exist. Can anyone come up with a clever trick that lets me decompress and print the last few lines of a compressed file without decompressing the entire thing?

  • 1
    I don't think you can... See also How can I tail a zipped file without reading its entire contents? and the links there... – don_crissti Jun 28 '16 at 10:23
  • @don_crissti well, dang! Looks like you're right. I'll leave this for a while in case anyone comes up with something really clever, and if not I'll copy one of the SO answers over. Probably the one stating that gzip doesn't support this. Feel free to do it yourself, by the way. – terdon Jun 28 '16 at 10:27
5

You can't, as it has been already said, if the files have been compressed with standard gzip. If you have control over the compression, you can use dictzip to compress the files, it compresses the files in separate blocks and you can decompress just the last block (typically 64KB). And it is backward compatible with gzip, meaning the dictzipped file is perfectly legal gzipped file as well.

Other possibility would be if you get the gzipped file as a concatenation of several already gzipped files, you could search for the last gzip signature and decompress everything after that.

  • I don't, sadly. I was hoping I could use the idea here to get the right chunks but I don't know if that's possible or what the chunks might be. – terdon Jun 28 '16 at 15:05
  • @terdon The idea there is to create the compressed file in chunks and then only uncompress the last chunk. If the compressed file has a single chunk, which is the case if you produced it with gzip, then “only uncompress the last chunk” = “uncompress the sole chunk” = “uncompress the whole file”. – Gilles Jun 28 '16 at 21:29
  • @Gilles I see. I had understood that the "chunks" were a result of the gzip algorithm and could perhaps be inferred from the compressed file. – terdon Jun 29 '16 at 8:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.