23

Is it possible to speed up the gzip process?

I'm using

mysqldump "$database_name" | gzip > $BACKUP_DIR/$database_name.sql.gz

to backup a database into a directory, $BACKUP_DIR.

the manpage says:

-# --fast --best
Regulate the speed of compression using the specified digit #, where -1 or --fast indi‐ cates the fastest compression method (less compression) and -9 or --best indicates the slowest compression method (best compression). The default compression level is -6 (that is, biased towards high compression at expense of speed).

  • How effective would it be to use --fast?
  • Is this effectively lowering the CPU usage on a modern computer?

My test results

I didn't notice any acceleration:

  • 7 min, 47 seconds (with default ratio -6)
  • 8 min, 36 seconds (with ratio --fast ( = 9 ))

So it seems it takes even longer to use the fast compression?

Only higher compression really slows it down:

  • 11 min, 57 seconds (with ratio --best ( = 1 ))

After getting the Idea with lzop I tested that too and it really is faster:

  • 6 min, 14 seconds with lzop -1 -f -o $BACKUP_DIR/$database_name.sql.lzo
6
  • Related: unix.stackexchange.com/questions/87111/…
    – slm
    Aug 29, 2013 at 16:58
  • Are you concerned about CPU time or overall (clock) time? --best is 9, --fast is 1. Aug 29, 2013 at 20:01
  • 3
    If gzip -1 is not fast enough, use lzop Aug 29, 2013 at 20:07
  • 1
    There is also lz4 which is even faster see this benchmark. But it seems lzop and lz4 need much more Memory. will it be a problem on my 1GB RAM single-core that lz4 uses 30 times as much memory on compression than gzip when I compress some large databases?
    – rubo77
    Aug 31, 2013 at 4:19
  • 1
    lz4 does not need that much memory. If you are using the lz4c utility, then almost all memory is allocated for I/O buffer. You can reduce that by using smaller buffers (option -B4 for 64KB buffers). Since smaller buffers also mean worse compression, it's useful to enable chained-block compression (option -BD). See : fastcompression.blogspot.fr/2013/08/…
    – Cyan
    Sep 2, 2013 at 11:50

3 Answers 3

30

If you have a multi-core machine using pigz is much faster than traditional gzip.

pigz, which stands for parallel implementation of gzip, is a fully functional replacement for gzip that exploits multiple processors and multiple cores to the hilt when compressing data. pigz was written by Mark Adler, and uses the zlib and pthread libraries.

Pigz ca be used as a drop-in replacement for gzip. Note than only the compression can be parallelised, not the decompression.

Using pigz the command line becomes

mysqldump "$database_name" | pigz > $BACKUP_DIR/$database_name.sql.gz
5
  • does this also speed up the process on a single-core?
    – rubo77
    Aug 29, 2013 at 16:44
  • @rubo77 - no this fans out the compression across multiple cores.
    – slm
    Aug 29, 2013 at 16:58
  • pigz increases the CPU usage but lowers the clock time it takes on multi-processors Aug 29, 2013 at 20:08
  • Indeed, and that's often what is desired. And usually (depending on the hardware) gzip compression is CPU bound (single-thread performance) and pigz compression is often IO bound.
    – Marco
    Aug 29, 2013 at 20:42
  • and you can use pigz -1 to use faster compression (and bigger file size), to increase speed even more if all your CPUs are maxed out on regular pigz Oct 17, 2021 at 23:26
7

From man gzip:

   -# --fast --best
          Regulate  the  speed  of compression using the
          specified digit #, where -1  or  --fast  indi‐
          cates  the  fastest  compression  method (less
          compression) and -9 or  --best  indicates  the
          slowest compression method (best compression).
          The default compression level is -6 (that  is,
          biased  towards high compression at expense of
          speed).
0
5

If you need it to be fast because of database locking issues, and you have a fast/large enough disk to hold the data uncompressed temporarily, you could consider using this method instead:

mysqldump "$database_name" > "$BACKUP_DIR"/"$database_name".sql
nice gzip "$BACKUP_DIR"/"$database_name".sql &

I.e. store the backup first (which is faster than gzipping it IF the disk is fast and the CPU is slow) and then have the gzipping occur in the background.

This might also allow you to use a better compression algorithm, as it no longer matters (directly) how long the compression takes.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .