1

My system is running Ubuntu 22.04.3 LTS. I plug a USB drive into that system. That USB drive (/dev/sdb) contains an Ubuntu installation, mostly in an ext4 partition (/dev/sdb3). That installation fills about 25GB of drive space.

With /dev/sdb3 unmounted, I use sudo zerofree -v /dev/sdb3 to zero free space on that USB drive. Then I use the following command to create a compressed image:

sudo dd if=/dev/sdb bs=16M conv=sync,noerror | pv | sudo pigz -c > /media/TargetDrive/UbuntuImage.dd.gz

This seems effective. The resulting .gz file weighs only 12GB.

Now I remove that USB drive (/dev/sdb) and replace it with a USB drive containing a YUMI exFAT installation.

When I seek information on this /dev/sdb drive using GParted, I get a warning - "Unable to read the contents of this file system!" - and an indication that exfatprogs is required for exfat file system support. I install exfatprogs and restart GParted. That eliminates that warning.

This YUMI exFAT installation fills 43GB on the USB drive. I understand that zerofree and Microsoft's sdelete -z E: don't zerofill on exFAT, so I follow advice that seems to recommend the following:

sudo mount -o rw /dev/sdbX /mnt
sudo dd if=/dev/zero of=/mnt/zero_file bs=32M
sudo sync
sudo rm /mnt/zero_file
sudo umount /dev/sdbX

Those commands duly produce and then remove a zero_file whose size does appear to fill empty space. (I saw but did not try this alternative, which was less clear to me: cat > /dev/sdbX/zeros < /dev/zero ; sync ; rm /dev/sdbX/zeros.)

Then I run substantially the dd command shown above. This time, dd and pigz do not achieve comparable compression. The 43GB exFAT installation is saved in an image file of 42GB.

Granted, the exFAT drive's contents are produced by Windows. But experience with Windows drive imaging software indicates that I should expect substantial compression.

My question: can the dd command be improved to yield better compression for this exFAT drive?

Note: I seem to have obtained similarly poor compression with an NTFS USB drive, but did not document each step in that case.

Improve this question
6
  • 1
    "can the dd command be improved to yield better compression […]?" – dd has nothing to do with compression. Compression depends on the entropy of the data and on pigz. By writing zeros you have reduced the entropy without touching useful data. What you can do is to replace pigz with something better. Your dd is responsible for corrupting data maybe. Do not use it. sudo pv /dev/sdb | … will read healthy /dev/sdb just fine. If you expect errors, use ddrescue.
    – Kamil Maciorowski
    Commented Sep 29, 2023 at 2:25
  • Cross-posted: askubuntu.com/questions/1477273/…
    – muru
    Commented Sep 29, 2023 at 3:02
  • I would recommend Clonezilla for this purpose. See this link.
    – sudodus
    Commented Sep 29, 2023 at 11:44
  • 1
    @sudodus : the Clonezilla suggestion is worthy but not ideal for me. I have commented and upvoted it. @KamilMaciorowski : sorry, I thought it was obvious that I was referring to the entire command beginning with dd. Comments on your link re: possible data corruption make the point: dd has been extensively researched and applied over decades. Being less thoroughly tested does not make an alternative superior. The conclusion appears to be simply that my dd command should use iflag=fullblock. My limited research suggests that pigz is superior - but I welcome a link to the contrary.
    – Ray Woodcock
    Commented Sep 29, 2023 at 17:27
  • 1
    @RayWoodcock, Making a dd image, I don't think you can improve the compression much by replacing pigz with some other tool (I use xz for similar purposes.) and you have already zeroized the unused data blocks (free space of the file system). But if the file system is encrypted, zeroizing and compression will not help much. If you backup the data on the file level when the file system is decrypted and mounted, it corresponds to skipping unused data blocks also when the file system is encrypted. I would use rsync or some dedicated backup tool for file level operations.
    – sudodus
    Commented Sep 29, 2023 at 18:57

1 Answer 1

Reset to default
2

(This didn't fit in a comment, so trying here.)

Excellent question. Unfortunately I'm not sure - that process is pretty much exactly what I would do. You don't say how big your "zeros" file is. If it's really small there just isn't any space available on the drive. The other alternative is that these disks are somehow automatically encrypted, in which case there won't be any actual zeroes written.

Another option, since you mentioned partitions, is that the partition you're writing to is small and there's a bunch of free space that you can't access but which you capture when you image the entire device. If that's the case, you either need to create a new filesystem that uses the rest of the space on the drive that you can write zeros to, OR write zeroes to the entire device (e.g, /dev/sdb, but be careful!) before you put the YUMI/etc. image on top of it.

Regarding this comment:

cat > /dev/sdbX/zeros < /dev/zero ; sync ; rm /dev/sdbX/zeros

That doesn't work. /dev/sdb is the entire device; /dev/sdbX is a partition on a device; /dev/sdb/whatever doesn't exist. The partition (or alternately, full device) needs a mounted filesystem in order to hold a file called zeros. That aside, the command is the same as cat /dev/zero > /path/to/zeros if that's clearer to you.

I would also suggest using bzip2 or xz in your pipeline if you want better compression. Those are both widely supported compression formats. The YUMI install image is probably already in a disk image that gets uncompressed when it boots so you'll get very little additional compression from imaging it.

Edit to add: zstd appears to be the currently-preferred compression algorithm. It has exceptional speed/size tradeoffs, and if time is no object also has some of the highest compression ratios. More information available here:

Improve this answer
3
  • I appreciate the correction re cat. My limited reading suggested that xz and bzip2 took dramatically longer for marginally improved compression. See e.g., reddit.com/r/linux/comments/3cnn54/… Regarding the YUMI compression, I'm not sure. YUMI doesn't seem to decompress anything, unlike Rufus. See raywoodcockslatest.wordpress.com/2023/09/25/ubuntu-bootable-iso/… Thanks again for your answer.
    – Ray Woodcock
    Commented Sep 29, 2023 at 18:03
  • 1
    Sorry, @RayWoodcock; I assumed YUMI was a distro until I skimmed your blog. Most "live distros" (that you run from a USB drive) are comprised of a Linux kernel and an initrd file that is an entire filesystem compressed into a single file (like what it sounds like you are trying to do, or what /dev/sdbX is). So compressing a system like this is near-fruitless. I understand 'xz' is a more modern p7zip (they are both LZMA) and you described generating a .xz file. Time or space is a tradeoff you have to choose. But I think I answered your main question. Let me know what you tried that didn't work.
    – rand'Chris
    Commented Sep 30, 2023 at 23:14
  • '@rand'Chris : that seems to be the case with YUMI too. A brief attempt to compress its contents using WinRAR reaches similar results: evidently compressed ISOs, forming most of its contents, being compressed by only 10-15%.
    – Ray Woodcock
    Commented Oct 7, 2023 at 0:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .