12

Is it possible to speed up the gzip process?

I'm using

mysqldump "$database_name" | gzip > $BACKUP_DIR/$database_name.sql.gz

to backup a database into a directory, $BACKUP_DIR.

the manpage says:

-# --fast --best
Regulate the speed of compression using the specified digit #, where -1 or --fast indi‐ cates the fastest compression method (less compression) and -9 or --best indicates the slowest compression method (best compression). The default compression level is -6 (that is, biased towards high compression at expense of speed).

  • How effective would it be to use --fast?
  • Is this effectively lowering the CPU usage on a modern computer?

My test results

I didn't notice any acceleration:

  • 7 min, 47 seconds (with default ratio -6)
  • 8 min, 36 seconds (with ratio --fast ( = 9 ))

So it seems it takes even longer to use the fast compression?

Only higher compression really slows it down:

  • 11 min, 57 seconds (with ratio --best ( = 1 ))

After getting the Idea with lzop I tested that too and it really is faster:

  • 6 min, 14 seconds with lzop -1 -f -o $BACKUP_DIR/$database_name.sql.lzo
  • Related: unix.stackexchange.com/questions/87111/… – slm Aug 29 '13 at 16:58
  • Are you concerned about CPU time or overall (clock) time? --best is 9, --fast is 1. – Stéphane Chazelas Aug 29 '13 at 20:01
  • 3
    If gzip -1 is not fast enough, use lzop – Stéphane Chazelas Aug 29 '13 at 20:07
  • 1
    There is also lz4 which is even faster see this benchmark. But it seems lzop and lz4 need much more Memory. will it be a problem on my 1GB RAM single-core that lz4 uses 30 times as much memory on compression than gzip when I compress some large databases? – rubo77 Aug 31 '13 at 4:19
  • lz4 does not need that much memory. If you are using the lz4c utility, then almost all memory is allocated for I/O buffer. You can reduce that by using smaller buffers (option -B4 for 64KB buffers). Since smaller buffers also mean worse compression, it's useful to enable chained-block compression (option -BD). See : fastcompression.blogspot.fr/2013/08/… – Cyan Sep 2 '13 at 11:50
16

If you have a multi-core machine using pigz is much faster than traditional gzip.

pigz, which stands for parallel implementation of gzip, is a fully functional replacement for gzip that exploits multiple processors and multiple cores to the hilt when compressing data. pigz was written by Mark Adler, and uses the zlib and pthread libraries.

Pigz ca be used as a drop-in replacement for gzip. Note than only the compression can be parallelised, not the decompression.

Using pigz the command line becomes

mysqldump "$database_name" | pigz > $BACKUP_DIR/$database_name.sql.gz
  • does this also speed up the process on a single-core? – rubo77 Aug 29 '13 at 16:44
  • @rubo77 - no this fans out the compression across multiple cores. – slm Aug 29 '13 at 16:58
  • pigz increases the CPU usage but lowers the clock time it takes on multi-processors – Stéphane Chazelas Aug 29 '13 at 20:08
  • Indeed, and that's often what is desired. And usually (depending on the hardware) gzip compression is CPU bound (single-thread performance) and pigz compression is often IO bound. – Marco Aug 29 '13 at 20:42
5

From man gzip:

   -# --fast --best
          Regulate  the  speed  of compression using the
          specified digit #, where -1  or  --fast  indi‐
          cates  the  fastest  compression  method (less
          compression) and -9 or  --best  indicates  the
          slowest compression method (best compression).
          The default compression level is -6 (that  is,
          biased  towards high compression at expense of
          speed).
3

If you need it to be fast because of database locking issues, and you have a fast/large enough disk to hold the data uncompressed temporarily, you could consider using this method instead:

mysqldump "$database_name" > "$BACKUP_DIR"/"$database_name".sql
nice gzip "$BACKUP_DIR"/"$database_name".sql &

I.e. store the backup first (which is faster than gzipping it IF the disk is fast and the CPU is slow) and then have the gzipping occur in the background.

This might also allow you to use a better compression algorithm, as it no longer matters (directly) how long the compression takes.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.