Skip to main content
73 votes

How to uncompress zlib data in UNIX?

I recommend pigz from Mark Adler, co-author of the zlib compression library. Execute pigz to see the available flags. You will notice: -z --zlib Compress to zlib (.zz) instead of gzip format. ...
snodnipper's user avatar
63 votes
Accepted

GZip doesn't produce the same compressed result on macOS vs Linux

Note that the compression algorithm (Deflate) in GZip is not strictly bijective. To elaborate: For some data, there's more than one possible compressed output depending on the algorithmic ...
schnaader's user avatar
  • 676
59 votes

Check validity of gz file

The gzip -t command only returns an exit code to the shell saying whether the file passed the integrity test or not. Example (in a script): if gzip -t file.gz; then echo 'file is ok' else ...
Kusalananda's user avatar
  • 338k
53 votes
Accepted

read first line from .gz compressed file without decompressing entire file

Piping zcat’s output to head -n 1 will decompress a small amount of data, guaranteed to be enough to show the first line, but typically no more than a few buffer-fulls (96 KiB in my experiments): ...
Stephen Kitt's user avatar
46 votes

7zip, xz, gzip, tar, etc -- what are the differences?

I first want to clarify that, of the list you provided, tar is the only one that is not a compression algorithm. tar is short for Tape Archive, and is used to create archive files. In short, a single ...
Peschke's user avatar
  • 4,166
36 votes
Accepted

How to turn a tar file to a tgz file?

A plain .tar archive created with cf (with or without v) is uncompressed; to get a .tar.gz or .tgz archive, compress it: gzip < my_files.tar > my_files.tgz You might want to add -9 for better ...
Stephen Kitt's user avatar
31 votes

Fastest and most efficient way to get number of records (lines) in a gzip-compressed file

The sed, perl and awk commands that you mention may be correct, but they all read the compressed data and counts newline characters in that. These newline characters have nothing to do with the ...
Kusalananda's user avatar
  • 338k
31 votes

How to gzip 100 GB files faster with high compression

If you are using gzip, you use mostly one processor core (well, some parts of the task, like reading and writing data are kernel tasks and kernel will use another core). Have a look at some multicore-...
d.c.'s user avatar
  • 887
30 votes
Accepted

Wget returning binary instead of html?

This is a gzip compressed file. You can find this out by running the file command, which figures out the file format from magic numbers in the data (this is how programs such as Text Wrangler figure ...
Gilles 'SO- stop being evil''s user avatar
30 votes

Is gzip atomic?

Is gzip atomic? No. It creates a compressed file and then removes the uncompressed original. Specifically, it does not compress a file in situ and there is a period of time while the file is being ...
Chris Davies's user avatar
27 votes
Accepted

How portable is a gzip file over 4 GB in size?

I have just noticed that gzip has a 4 GB size limit More accurately, the gzip format can’t correctly store uncompressed file sizes over 4GiB; it stores the lower 32 bits of the uncompressed size, and ...
Stephen Kitt's user avatar
25 votes
Accepted

Uncompressed file estimation wrong?

This is caused by the size of the field used to store the uncompressed size in gzipped files: it’s only 32 bits, so gzip can only store sizes of files up to 4 GiB. Anything larger is compressed and ...
Stephen Kitt's user avatar
23 votes

Can I zip an entire folder using gzip?

The Answer to the question “Can I zip an entire folder using gzip [on linux]?” is that you can use the zip program in place of gzip, with the syntax: zip -r <zip file name> <folder to zip>
ize's user avatar
  • 259
23 votes
Accepted

Print archive file list instantly (without decompressing entire archive)

It's important to understand there's a trade-off here. tar means tape archiver. On a tape, you do mostly sequential reading and writing. Tapes are rarely used nowadays, but tar is still used for its ...
Stéphane Chazelas's user avatar
22 votes

Why are tar archive formats switching to xz compression to replace bzip2 and what about gzip?

LZMA2 is a block compression system whereas gzip is not. This means that LZMA2 lends itself to multi-threading. Also, if corruption occurs in an archive, you can generally recover data from subsequent ...
Mark Warburton's user avatar
22 votes

Fastest and most efficient way to get number of records (lines) in a gzip-compressed file

Use unpigz. Kusalananda's answer is correct, you will need to uncompress that entire file to scan its contents. /bin/gunzip does this as fast as it can, on a single core. Pigz is a parallel ...
marcelm's user avatar
  • 2,485
21 votes

why should I use tar.xz instead of tar.gz? xz is a lossless data compression program and file format

gzip and xz uses two different algorithms, and therefore they perform differently, both in terms of what level of compression they achieve and in terms of the amount of resources they consume while ...
Kusalananda's user avatar
  • 338k
20 votes

On-the-fly stream compression that doesn't spill over into hardware resources?

dd reads and writes data one block at a time, and it only ever has one block outstanding. So valgrind dd if=/dev/zero status=progress of=/dev/null bs=1M shows that dd uses approximately 1MB of ...
Stephen Kitt's user avatar
20 votes

GZip doesn't produce the same compressed result on macOS vs Linux

The format should be very stable, but see its description. It contains a field for operating system ID.  Obviously that may differ for macOS and Linux and FreeBSD and ...
vonbrand's user avatar
  • 18.3k
20 votes

How to gzip 100 GB files faster with high compression

We have 100+ GB files and while trying to perform gzip using below command , gzip is taking minimum 1-2 hours to get complete With CPU (taken from a comment): Intel® Core™ i3-2350M @ 2.30GHz, which ...
Vlastimil Burián's user avatar
19 votes

How to add/update a file to an existing tar.gz archive?

Tar was originally meant to be used for tapes, so "replacing" files is not really in the design. However, you can use --delete to delete the file from the original tar and then -u to update ...
rleibman's user avatar
  • 191
18 votes
Accepted

On-the-fly stream compression that doesn't spill over into hardware resources?

Technically you don't even need dd: gzip < /dev/drive > drive.img.gz If you do use dd, you should always go with larger than default blocksize like dd bs=1M or suffer the syscall hell (dd's ...
frostschutz's user avatar
  • 49.5k
17 votes

Create an archive with command "gzip"

gzip works with files, not directories. Therefore to create an archive of a directory you need to use tar (which will create a single tarball out of multiple files): tar cvf myetc.tar /etc or, for ...
dr_'s user avatar
  • 30k
15 votes

Wget returning binary instead of html?

This file is still compressed with gzip. You can see that the first two bytes 0x1f8b match the gzip signature. So to read the data you need to uncompress it. mv file.dat file.dat.gz gunzip file.dat....
Chris Davies's user avatar
15 votes
Accepted

Why does a gunzip to dd pipeline slow down at the end?

I figure there is a cache somewhere that is holding up the final 5MB but I thought fsync should make sure that doesn't happen conv=fsync means to write back any caches by calling fsync - after dd has ...
sourcejedi's user avatar
15 votes

Is gzip atomic?

It's not atomic (the Unix filesystem API doesn't really provide any way to perform atomic operations that affect multiple files), but it is fail-safe. The compressed file is a new file, it doesn't ...
Barmar's user avatar
  • 9,956
13 votes

How to uncompress zlib data in UNIX?

On macOS, which is a full POSIX compliant UNIX (formally certified!), OpenSSL has no zlib support, there is no zlib-flate either and while the first solution works as well as all the Python solutions, ...
Mecki's user avatar
  • 296
13 votes

Is there a tool that combines zcat and cat transparently?

I use exactly for the same purpose: { cat /var/log/messages ; zcat /var/log/messages*.gz ; }| grep something | grep "something else" ....
Washuu's user avatar
  • 151
12 votes
Accepted

gzip: unexpected end of file with - how to read file anyway

Apart from the very end of the file, you will be able to see the uncompressed data with zcat (or gzip -dc, or gunzip -c): zcat log.gz | tail or zcat log.gz | less or zless log.gz gzip will do ...
Kusalananda's user avatar
  • 338k

Only top scored, non community-wiki answers of a minimum length are eligible