I currently have a compressed file, A.gz which contains lots of tabulated data, including a header in the first line. I want to create another file, B.gz, which has the same data as the previous file, but with a different header.

The simple way to do this would be decompressing all of A.gz, tail-ing everything but the first line, and re-compressing everything. However, this seems terrible inefficient, specially because the concatenation of two gzip-ed files decompresses correctly to the concatenation of the decompressed versions.

I was wondering if there was a way to do this similar to this:

zcat A.gz | head -n 1 | process_header | gzip > B.gz
cat A.gz | (remove compressed header) >> B.gz

Without having to decompress all of A.gz.


If you just wanted to insert another line on top, it would be simple.

echo some line | gzip > newfile.gz
cat newfile.gz oldfile.gz > result.gz

gzip allows concatenation. If you don't mind it reporting a wrong uncompressed filesize if you just look at the file w/o uncompressing it, that is. Also some programs can not handle such files, WinRAR for example.

To get closer to what you actually want, the question is whether your gzip file is made up of blocks that function entirely independent from one another, and if so, how to find the block boundary.

If you knew you wanted to do this beforehand and created the gzip by concatenating two independent gzip files in the first place, it would be easy to solve; however on arbitrary gzip files, if it can be done at all, it would require more in depth knowledge of the gzip file format.

I remember there was such a program for bzip2 (but I forgot its name), it created a bzip2 block map that would allow you direct access to specific offsets without uncompressing everything that came before it.

On the bottom line, though, most people just recompress. You likely won't be able to avoid re-writing the entire file anyhow and writing files is usually slower than gzip can compress data, so - if you managed to pull it off, you'd probably save some CPU cycles, but no time.

Not a solution to your gzip question but... don't use tail to get rid of the first line, it's probably very inefficient compared to a sed 1d or whatever. No need to count all lines of a file just to get rid of the first one.

  • 1
    You don't need to count all lines when tailing something. Just run tail -n +2 file to remove the header of a file with an arbitrary amount of lines. – Martín Fixman Sep 1 '15 at 17:39

How about

zcat A.gz | awk '{if(NR==1){print "myheader"}else{print $0}}' | gzip > B.gz

If NR (record number) is 1, output your very own header. Leave all other lines intact.

  • That's exactly what I'm trying to avoid. Even if awk doesn't parse all the lines, they are unnecessarily decompressed and re-compressed into B.gz. – Martín Fixman Sep 1 '15 at 15:13

!!! This is only a thought !!!

You can try and run

zcat file | head -n100 > tempfile 
vim tempfile # edit the file header
cat tempfile | gzip | dd of=B.gz conv=notrunc

this will extract only the first 100 lines from the compressed files, and then re-compress them and finely will overwrite the same blocks on file B.gz e.

the problem is, that is not a real solution because you'll need to make sure that the BEFORE and AFTER data consume the same amount of bytes, and then run over the file and calculate the CRC32 for the new file and write it to the footer of the file.

You probably better off with the answer steve give you.

  • That won't work, even if the modified part happens to use the same number of bytes. The interpretation of bytes in a compressed stream depends on the previous bytes — that's how compression works, by exploiting resemblance between parts of the data. If you concatenate segments of compressed data, you'll get garbage. Fortunately gunzip will helpfully tell you that the checksum is wrong (with high probability). – Gilles 'SO- stop being evil' Sep 1 '15 at 22:32
  • I know, but if you take enough bytes from the header, the probability that the letter bytes will eventually correlate, as you only changing a small part of the header and the data stay the same. and the "compression table" will always be based on the latest data. but again not a practical solution. – Rabin Sep 2 '15 at 7:42
  • Calculating the probabilty goes beyond my knowledge, but I think the probability to have a match would be negligible, and goes down when the header becomes larger up to a very large point where you run into the compressor's memory limits. – Gilles 'SO- stop being evil' Sep 2 '15 at 8:38

Still decompressing it, but much quicker than zcat or gzip for large files:

pigz -dc new_header.txt.gz A.gz | sed '2d' | pigz > B.gz

Just put your new header in new_header.txt (without a newline) and gzip it before running the above.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.