32

Once a file is gzipped, is there a way of quickly querying it to say what the uncompressed file size is (without decompressing it), especially in cases where the uncompressed file is > 4GB in size.

According to the RFC https://www.rfc-editor.org/rfc/rfc1952#page-5 you can query the last 4 bytes of the file, but if the uncompressed file was > 4GB then the value just represents the uncompressed value modulo 2^32

This value can also be retrieved by running gunzip -l foo.gz, however the "uncompressed" column just contains uncompressed value modulo 2^32 again, presumably as it's reading the footer as described above.

I was just wondering if there is a way of getting the uncompressed file size without having to decompress it first, this would be especially useful in the case where gzipped files contain 50GB+ of data and would take a while to decompress using methods like gzcat foo.gz | wc -c


EDIT: The 4GB limitation is openly acknowledged in the man page of the gzip utility included with OSX (Apple gzip 242)

  BUGS
    According to RFC 1952, the recorded file size is stored in a 32-bit
    integer, therefore, it can not represent files larger than 4GB. This
    limitation also applies to -l option of gzip utility.
4
  • 2
    +1 good question! I suspect the answer is no, that header format was designed in a time before such file sizes were anticipated. Thinking about it, gzip must be older than many users in this community!
    – Celada
    Feb 7, 2015 at 10:32
  • 2
    gzip came out in 1992. I would be surprised if there were many 23 year olds roaming around here. I'm sure there are some but from what I can tell the median age is around 30-35.
    – Bratchley
    Feb 7, 2015 at 12:47
  • 2
    May be a good time to switch to xz which doesn't have that limitation. GNU is switching to xz. Feb 7, 2015 at 14:52
  • @StéphaneChazelas Interesting. Unfortunately the files I'm interested in are outside of my control (i.e. we receive them compressed), but it certainly looks like xz would 'solve` this issue.
    – djhworld
    Feb 7, 2015 at 19:53

5 Answers 5

12

I believe the fastest way is to modify gzip so that testing in verbose mode outputs the number of bytes decompressed; on my system, with a 7761108684-byte file, I get

% time gzip -tv test.gz
test.gz:     OK (7761108684 bytes)
gzip -tv test.gz  44.19s user 0.79s system 100% cpu 44.919 total

% time zcat test.gz| wc -c
7761108684
zcat test.gz  45.51s user 1.54s system 100% cpu 46.987 total
wc -c  0.09s user 1.46s system 3% cpu 46.987 total

To modify gzip (1.6, as available in Debian), the patch is as follows:

--- a/gzip.c
+++ b/gzip.c
@@ -61,6 +61,7 @@
 #include <stdbool.h>
 #include <sys/stat.h>
 #include <errno.h>
+#include <inttypes.h>
 
 #include "closein.h"
 #include "tailor.h"
@@ -694,7 +695,7 @@
 
     if (verbose) {
         if (test) {
-            fprintf(stderr, " OK\n");
+            fprintf(stderr, " OK (%jd bytes)\n", (intmax_t) bytes_out);
 
         } else if (!decompress) {
             display_ratio(bytes_in-(bytes_out-header_bytes), bytes_in, stderr);
@@ -901,7 +902,7 @@
     /* Display statistics */
     if(verbose) {
         if (test) {
-            fprintf(stderr, " OK");
+            fprintf(stderr, " OK (%jd bytes)", (intmax_t) bytes_out);
         } else if (decompress) {
             display_ratio(bytes_out-(bytes_in-header_bytes), bytes_out,stderr);
         } else {

A similar approach has been implemented in gzip, and will be included in the release following 1.11; gzip -l now decompresses the data to determine its size.

6
  • Does it still build the actual data internally, or is -t already optimized in that regard? The improvement is small enough to make it look like you've only saved the output time. Feb 7, 2015 at 15:03
  • Yes, it needs to decompress everything to figure out the original size... So this is only saving the output time, but I think that's all that can be saved. Feb 7, 2015 at 15:16
  • Interesting, yeah I was thinking you would need to change code to actually get this to work. Unfortunately in my instance the files I'm interested are not actually in my control, I receive them from an external party so wouldn't be able to compress them in the first place. I think the only way to fully support >4GB files would be to patch gzip to have a 12 byte footer, 4 bytes for CRC and 8 bytes (64 bit) for the filesize. However this would break backwards compatibility with existing gzips!
    – djhworld
    Feb 7, 2015 at 19:49
  • The solution I give above doesn't involve compressing the files initially, even though I'm running gzip; I just run gzip on the compressed files, which doesn't re-compress them, it just verifies them. (The patch is a quick-and-dirty proof-of-concept, it needs a few more changes to work with gunzip.) Feb 7, 2015 at 19:51
  • @StephenKitt Ah interesting! An even better/dirtier hack would be to embed that data in the FCOMMENT field. That way users could query a byte range to retrieve that data. This would be useful in my case, especially for items stored in Amazon S3
    – djhworld
    Feb 7, 2015 at 20:10
1

The gzip format stores the uncompressed size in only 4 bytes (the last 4 bytes of the file), so the uncompressed size stored is actually the size modulo 2**32 (4GiB). In cases when the real uncompressed size is less than 4GiB the value will be correct but for uncompressed files greater than 4GiB the only way to have the correct value is by reading the whole file. But it's possible to estimate it (Python code below)!

If the compressed size is greater than the uncompressed, probably the uncompressed is greater than 4GiB and we try to guess the correct size by adding "1" bits to the left until the new size is greater than the compressed one and greater than 4GiB. Note that this guess may be wrong for 2 reasons:

  • The compressed size may be greater than the uncompressed one in some cases (like trying to compress an already compressed file); or
  • For very big files we keep shifting the bit "1" to the left several times, which makes a "hole" between the digit "1" and the original 32 bits (e.g.: shifting 5 times lead to in 10000X, where X are the original 32 bits). The value returned is the minimum expected size for the uncompressed file, since there's no way to correctly "fill the hole" without reading the whole file.

Here is a Python code to estimate the uncompressed size from my rows project:

import os
import struct

def estimate_gzip_uncompressed_size(filename):
    compressed_size = os.stat(filename).st_size
    with open(filename, mode="rb") as fobj:
        fobj.seek(-4, 2)
        uncompressed_size = struct.unpack("<I", fobj.read())[0]
    if compressed_size > uncompressed_size:
        i, value = 32, uncompressed_size
        while value <= 2**32 and value < compressed_size:
            value = (1 << i) ^ uncompressed_size
            i += 1
        uncompressed_size = value
    return uncompressed_size

I had a bad time implementing a progress bar to report the importing of a gzipped CSV file to PostgreSQL on rows pgimport, so I've written the function above to estimate the real size (the program will know if the estimated value is wrong, since it's going to read the whole file, then it just updates the progress bar with the correct "new" value).

Note: using gzip --list <filename> to get the uncompressed size is not an option for me because:

  • Prior to version 2.12, the command run quickly but reported the wrong uncompressed size (it just reads the 4 last bytes); and
  • Version 2.12 fixed this bug by reading the whole file (just to print the uncompressed size!) - it's not an option, since the file is big and it'll take a lot of time. From the 2.12 release notes:

'gzip -l' no longer misreports file lengths 4 GiB and larger. Previously, 'gzip -l' output the 32-bit value stored in the gzip header even though that is the uncompressed length modulo 2**32. Now, 'gzip -l' calculates the uncompressed length by decompressing the data and counting the resulting bytes. Although this can take much more time, nowadays the correctness pros seem to outweigh the performance cons.

3
  • 1
    Unfortunately there’s no sensible way to estimate the uncompressed size of a gzipped file larger than 4GiB. Your approach (like any other, I’m not bashing your attempt specifically) has no way of figuring out how close its result is without decompressing all the data... Most of the large gzipped files I have are less than 4GiB in size when compressed, so your estimate always ends up being (4GiB + the compressed size), which is usually wrong (in my tests) by an order of magnitude. Jun 27 at 15:04
  • 1
    Your heuristic also ends up giving an incorrect result if the compressed file is smaller than the truncated uncompressed size, which can happen if the original file compresses well and has a size close to, for example, 7GiB. Jun 27 at 15:06
  • @StephenKitt you're right. In my case I just wanted to have a best guess during the file decompression process, so I have many chances to fix the uncompressed size while the command is running (the actual implementation on rows CLI have the behavior I expected). Jun 28 at 2:08
0

If you need the size of a compressed file or set of files, your best bet is to use tar -z or tar -j instead of gzip as tar includes the uncompressed files size. Use lesspipe to peek at the list of files:

aptitude install lesspipe
lesspipe <compressed file> | less

If less is configured to use lesspipe:

less <compressed file>

Just bear in mind it can take a very long time though. However your system remains responsive, which allows you to kill the decompression process.

Another approach would be to log the compressed ratio and query that [text] file instead:

gzip --verbose file 2>&1 | tee file.gz.log
file:    64.5% -- replaced with file.gz

It requires computation to find the real file size though.

You could also do the same with tar, which is in fact what I do with big sized backups as it prevents running through the whole decompression process to only get a file size or name, for instance.

4
  • 2
    Doesn't tar.gz have to be decompressed completely as well to get the list of all files? Feb 7, 2015 at 11:04
  • Indeed it has to be. This is the only way I can think of to get the uncompressed file size. With tar you have the original file size logged in the archive. I'm not sure zip behaves differently, on the other hand.
    – user86969
    Feb 7, 2015 at 11:08
  • 1
    At that point the OP might as well do the wc -c command.
    – Bratchley
    Feb 7, 2015 at 12:34
  • @Bratchley of course. But it'd take a considerable amount of time to get all the results. Hence my two suggestions to log file sizes.
    – user86969
    Feb 7, 2015 at 12:59
0

What about

gzip -l file.gz|tail -n1|awk '{print $2}'

numfmt --to=iec $(gzip -l file.gz|tail -n1|awk '{print $2}')
1
  • 4
    That doesn’t work for large files, as explained by the OP. Jun 8, 2018 at 12:17
-3
gunzip -c $file | wc -c

This will take a long time, but will give you the final size in bytes.

1
  • 7
    This is exactly what the OP is trying to avoid having to do.
    – depquid
    Apr 20, 2016 at 15:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.