22

Once a file is gzipped, is there a way of quickly querying it to say what the uncompressed file size is (without decompressing it), especially in cases where the uncompressed file is > 4GB in size.

According to the RFC https://tools.ietf.org/html/rfc1952#page-5 you can query the last 4 bytes of the file, but if the uncompressed file was > 4GB then the value just represents the uncompressed value modulo 2^32

This value can also be retrieved by running gunzip -l foo.gz, however the "uncompressed" column just contains uncompressed value modulo 2^32 again, presumably as it's reading the footer as described above.

I was just wondering if there is a way of getting the uncompressed file size without having to decompress it first, this would be especially useful in the case where gzipped files contain 50GB+ of data and would take a while to decompress using methods like gzcat foo.gz | wc -c


EDIT: The 4GB limitation is openly acknowledged in the man page of the gzip utility included with OSX (Apple gzip 242)

  BUGS
    According to RFC 1952, the recorded file size is stored in a 32-bit
    integer, therefore, it can not represent files larger than 4GB. This
    limitation also applies to -l option of gzip utility.
  • 2
    +1 good question! I suspect the answer is no, that header format was designed in a time before such file sizes were anticipated. Thinking about it, gzip must be older than many users in this community! – Celada Feb 7 '15 at 10:32
  • 2
    gzip came out in 1992. I would be surprised if there were many 23 year olds roaming around here. I'm sure there are some but from what I can tell the median age is around 30-35. – Bratchley Feb 7 '15 at 12:47
  • 2
    May be a good time to switch to xz which doesn't have that limitation. GNU is switching to xz. – Stéphane Chazelas Feb 7 '15 at 14:52
  • @StéphaneChazelas Interesting. Unfortunately the files I'm interested in are outside of my control (i.e. we receive them compressed), but it certainly looks like xz would 'solve` this issue. – djhworld Feb 7 '15 at 19:53
11

I believe the fastest way is to modify gzip so that testing in verbose mode outputs the number of bytes decompressed; on my system, with a 7761108684-byte file, I get

% time gzip -tv test.gz
test.gz:     OK (7761108684 bytes)
gzip -tv test.gz  44.19s user 0.79s system 100% cpu 44.919 total

% time zcat test.gz| wc -c
7761108684
zcat test.gz  45.51s user 1.54s system 100% cpu 46.987 total
wc -c  0.09s user 1.46s system 3% cpu 46.987 total

To modify gzip (1.6, as available in Debian), the patch is as follows:

--- a/gzip.c
+++ b/gzip.c
@@ -61,6 +61,7 @@
 #include <stdbool.h>
 #include <sys/stat.h>
 #include <errno.h>
+#include <inttypes.h>

 #include "closein.h"
 #include "tailor.h"
@@ -694,7 +695,7 @@

     if (verbose) {
         if (test) {
-            fprintf(stderr, " OK\n");
+            fprintf(stderr, " OK (%jd bytes)\n", (intmax_t) bytes_out);

         } else if (!decompress) {
             display_ratio(bytes_in-(bytes_out-header_bytes), bytes_in, stderr);
@@ -901,7 +902,7 @@
     /* Display statistics */
     if(verbose) {
         if (test) {
-            fprintf(stderr, " OK");
+            fprintf(stderr, " OK (%jd bytes)", (intmax_t) bytes_out);
         } else if (decompress) {
             display_ratio(bytes_out-(bytes_in-header_bytes), bytes_out,stderr);
         } else {
  • Does it still build the actual data internally, or is -t already optimized in that regard? The improvement is small enough to make it look like you've only saved the output time. – frostschutz Feb 7 '15 at 15:03
  • Yes, it needs to decompress everything to figure out the original size... So this is only saving the output time, but I think that's all that can be saved. – Stephen Kitt Feb 7 '15 at 15:16
  • Interesting, yeah I was thinking you would need to change code to actually get this to work. Unfortunately in my instance the files I'm interested are not actually in my control, I receive them from an external party so wouldn't be able to compress them in the first place. I think the only way to fully support >4GB files would be to patch gzip to have a 12 byte footer, 4 bytes for CRC and 8 bytes (64 bit) for the filesize. However this would break backwards compatibility with existing gzips! – djhworld Feb 7 '15 at 19:49
  • The solution I give above doesn't involve compressing the files initially, even though I'm running gzip; I just run gzip on the compressed files, which doesn't re-compress them, it just verifies them. (The patch is a quick-and-dirty proof-of-concept, it needs a few more changes to work with gunzip.) – Stephen Kitt Feb 7 '15 at 19:51
  • @StephenKitt Ah interesting! An even better/dirtier hack would be to embed that data in the FCOMMENT field. That way users could query a byte range to retrieve that data. This would be useful in my case, especially for items stored in Amazon S3 – djhworld Feb 7 '15 at 20:10
0

If you need the size of a compressed file or set of files, your best bet is to use tar -z or tar -j instead of gzip as tar includes the uncompressed files size. Use lesspipe to peek at the list of files:

aptitude install lesspipe
lesspipe <compressed file> | less

If less is configured to use lesspipe:

less <compressed file>

Just bear in mind it can take a very long time though. However your system remains responsive, which allows you to kill the decompression process.

Another approach would be to log the compressed ratio and query that [text] file instead:

gzip --verbose file 2>&1 | tee file.gz.log
file:    64.5% -- replaced with file.gz

It requires computation to find the real file size though.

You could also do the same with tar, which is in fact what I do with big sized backups as it prevents running through the whole decompression process to only get a file size or name, for instance.

  • 2
    Doesn't tar.gz have to be decompressed completely as well to get the list of all files? – frostschutz Feb 7 '15 at 11:04
  • Indeed it has to be. This is the only way I can think of to get the uncompressed file size. With tar you have the original file size logged in the archive. I'm not sure zip behaves differently, on the other hand. – user86969 Feb 7 '15 at 11:08
  • 1
    At that point the OP might as well do the wc -c command. – Bratchley Feb 7 '15 at 12:34
  • @Bratchley of course. But it'd take a considerable amount of time to get all the results. Hence my two suggestions to log file sizes. – user86969 Feb 7 '15 at 12:59
0

What about

gzip -l file.gz|tail -n1|awk '{print $2}'

numfmt --to=iec $(gzip -l file.gz|tail -n1|awk '{print $2}')
  • That doesn’t work for large files, as explained by the OP. – Stephen Kitt Jun 8 '18 at 12:17
-2
gunzip -c $file | wc -c

This will take a long time, but will give you the final size in bytes.

  • 5
    This is exactly what the OP is trying to avoid having to do. – depquid Apr 20 '16 at 15:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.