8

I often work on full disk images, I often need to read and write full disk images on not so capable laptops using a USB-3 disk as my temporary medium.

Duplicating a raw image is probably the fastest way, but I have to deal with limited storage space available.

What I need is high throughput and low CPU usage - compression ratio is not that important for me.

I'd like to have a tool that can very lightly compress the images created - mostly encoding empty space on the disk so that I can only store useful data, while making it as fast as possible - hopefully almost as fast as a raw dd duplication.

I tied to use pxz -1, as it can use multiple CPU cores to speed up the compression, hoping for the fastest compression to still be useful. That doesn't however seem to be as fat as I'd like.

Compressing a 500 GB disk image was estimated to take 21 hours:

mint Backup # dd if=/dev/sda bs=100M | pv -ptera -s500G | pxz -1 > Lenovo-Win8-sda.dd.xz
0:09:21 [9.26MiB/s] [6.54MiB/s] [>            ]  0% ETA 21:34:46

No compression at all predicted 3 hours to complete the imaging:

mint Backup # dd if=/dev/sda bs=100M | pv -ptera -s500G > Lenovo-Win8-sda.dd
0:00:20 [80.2MiB/s] [49.8MiB/s] [>           ]  0% ETA 2:50:20

Using lbzip2 -1 for compresson seems to be slightly faster than pxz -1 with ETA at 15 hours, which is still very slow.

mint Backup # dd if=/dev/sda bs=100M | pv -ptera -s500G | lbzip2 -1 > Lenovo-Win8-sda.dd.bz2
0:00:22 [9.07MiB/s] [9.76MiB/s] [>             ]  0% ETA 14:33:38

Are there any faster compression tools for Linux?

Improve this question
4
  • 2
    See pixz -1 (multi-threaded), xz -1 -T (recent versions of xz-utils), pigz -1, lzop, lz4 (single threaded but still faster than the *xz one on a 16 core system here, not as good compression ratio though).
    – Stéphane Chazelas
    Commented Jun 6, 2018 at 14:34
  • 4
    With pixz/pxz/xz -T, that means you can mount your disk image without having to decompress it (using nbdkit) which is a huge advantage that you may want to take into consideration.
    – Stéphane Chazelas
    Commented Jun 6, 2018 at 14:39
  • Whoa - mounting a compressed disk image can be a huge thing. I wonder if lz4 supports something similar.
    – unfa
    Commented Jun 6, 2018 at 14:41
  • It's possible in theory as long as you can compress separate blocks independently, however nbdkit only supports xz and gzip (and I suspect for gzip it decompresses the whole image fully instead of individual blocks). You could also use a compressed qcow2 image (not fast though as qemu-img convert does a single-thread gzip; still faster than xz on a single-core machine).
    – Stéphane Chazelas
    Commented Jun 8, 2018 at 13:34

4 Answers 4

Reset to default
9

LZ4

apt install liblz4-tool

Then you can compress with lz4 command and decompress with lz4 -d.

It defaults to the fastest compression mode.

500 GB from internal hard drive to an external drive over USB 3.0 is estimated to take between 2 and 3 hours:

mint Backup # dd if=/dev/sda bs=100M | pv -ptera -s500G | lz4 > Lenovo-Win8-sda.dd.lz4
0:02:47 [97.5MiB/s] [58.1MiB/s] [>                ]  1% ETA 2:24:11
Improve this answer
1
  • Yes, in my tests, it's also the fastest I've found even though it's not multithreaded. About as fast as lzop but slightly better compression ratio.
    – Stéphane Chazelas
    Commented Jun 6, 2018 at 14:42
7

So, this won't be the absolute fastest (It's slower than LZ4, LZOP, and Snappy), but might be fast enough for your usage and will get way better ratios than LZ4 (and transferring less data will save you time too).

ZStandard

Official website: https://facebook.github.io/zstd/

The Debian package is called zstd

Unlike LZ4, it's multi-threaded (for both compression and decompression), and with the lowest compression settings it can easily saturate a USB 3.0 link (which your test with LZ4 is probably already doing), while getting compression ratios on par with the default settings in Gzip.

Improve this answer
3

This sounds like an XY Problem.  A generic compression tool isn’t particularly likely to do a good job on a disk image, because unused space is not guaranteed to be empty.  (Admittedly, there are ways to zero out unused space in a filesystem.)

You might be better off using a backup tool — such as rsync, Clonezilla, Time Machine, Duplicity, dump, or even tar, to name a few — that understands the filesystem format and backs up only the portions that are used.  See this question for a discussion of some of them.  Such an approach would have the advantage that it becomes much easier to restore the backup to a disk of a different size.

Improve this answer
3
  • 1
    I know I should wash the empty space on partitions before imaging the disk to save space - however washing the filesystems is also time-consuming. As I most often than not work with bootable disk images, and I need to preserve the MBR and the exact partition layout so I can restore a hard disk to it's exact state later - so working on file level won't cut it for me unfortunately.
    – unfa
    Commented Jun 7, 2018 at 8:28
  • 1
    I don't believe the existence of the so-named "xy problem". The OP wants to compress disk images, you can say that it is sub-optimal because it doesn't happen on the FS level, and you are right. But the question is still valid, with valid - although suboptimal - solutions.
    – peterh
    Commented Jun 7, 2018 at 9:45
  • 2
    take care when choosing between file-system snapshots and disk images for backups, the file-system and the storage media do different jobs and backing up one or the other has different capabilities and limitations
    – ThorSummoner
    Commented Feb 17, 2021 at 19:48
1

I use Clonezilla and it can create cloned copies or compressed images of a whole drive including the head of the drive with boot sector for BIOS mode, partition table and everything.

A Clonezilla image is a directory with a number of files, where the big files are compressed, and Clonezilla can restore from such an image to a drive of exactly the same size or bigger (but not to a smaller drive). Such a drive behaves like the original one (is bootable etc). If the drive is bigger, and the partition table is GPT, you may have to restore the backup partition table at the tail end of the drive, manually with gdisk or with the script gpt-fix.

Clonezilla is smart enough to identify used blocks and skip free blocks in file systems (I have used it with several linux file systems and Microsoft file systems, where it uses partclone), so if there is a fair pencentage of free blocks, it will be much faster that 'raw' cloning and compression (with or without zeroed out free space).

When I want high compression and very flexible usage without the end user needing Clonezilla, I zero out free space and pipe dd via xz to get an img.xz file. But this process is very slow, and not suitable for your case.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .