19

Is it possible to compress a very large file (~30 GB) using gzip? If so, what commands, switches, and options should I use?

Or is there another program (preferably one commonly available on Ubuntu distributions) that I can use to compress/zip very large files? Do you have any experience with this?

1
  • Most hardcore compressor is xz, or its parallel version, pxz. pxz -9ve - <in >out.xz and you will bake a chicken on your PC, but the result will be around 60% of the gzip result. You need around 7 GB for that.
    – peterh
    May 8 '17 at 21:54
23

AFAIK there is no limit of size for gzip - at least not 30GB. Of course, you need the space for the zipped file on your disc, both versions will be there simultanously while compressing.

bzip2 compresses files (not only big ones :-) better, but it is (sometimes a lot) slower.

7
  • 5
    bzip2 is especially good at compressing source code (and the like). It is a good competitor to gzip for generic use too. Availability for gzip across different operating systems is better than bzip2
    – jippie
    Jun 3 '12 at 19:20
  • 3
    @Andrew Yes, gzip/bzip2/etc... are all lossless.
    – Renan
    Jun 3 '12 at 20:06
  • 1
    And there is pbzip, too - that will use more than one CPU-core. But still way slower than gzip.
    – Nils
    Jun 10 '12 at 20:45
  • 1
    @Nils: on my Ubuntu there is only pbzip2
    – rubo77
    Aug 29 '13 at 16:31
  • 1
    If you want a higher-compression alternative to gzip, use xz. It's better, faster (and much faster to decompress), and generally recommended. If you want a lower-compression faster alternative to gzip, there's lzop / lz4. (xz -0 is actually a good choice for medium speed.) Aug 11 '15 at 19:40
9

if you need a good compresion rate, you can try lzma. it is faster and more efficient than bzip2 and may be faster even than gzip (i don't know that for sure)

http://www.thegeekstuff.com/2010/06/lzma-better-compression-than-bzip2-on-unix-linux/

1
  • 5
    lzma is deprecated in favour of xz now. Same algorithm, somewhat different (improved?) file format wrapped around it. LZMA is slower than gzip, but at max speed it's still fairly good compression on highly redundant stuff like JSON data. (xz -0) Aug 11 '15 at 19:41
4

If you are running into the limit, rearrange. Instead of:

gzip file

do:

gzip < file > file.gz

Works just fine.

2

The gzip format represents the input size modulo 2^32, so the --list option reports incorrect uncompressed sizes and compression ratios for uncompressed files 4 GB and larger.

So, take bzip2 (v1.0.2 and upwards) or xz.

3
  • If it should precisely be gzip only, then your file has to be divided into smaller parts up to 4 Gb each.
    – cuneiform
    Sep 11 '14 at 7:29
  • There are file splitters available for this.
    – Vérace
    Mar 22 '16 at 5:23
  • Well, it doesn't have to be split up, it's just that the reported sizes will be wrong.
    – Law29
    Mar 22 '16 at 7:05
2

XZ set to lowest compression may well beat gzip for speed and compression.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.