6

On my Linux Mint 17.2 VMware guest, df -h was reporting my total disk usage steady at about 10GB total. I'm using this machine for Ruby on Rails development inside a Windows host running Workstation 12.1.1 Pro.

The *.vmdk files kept on steadily growing to approx 100GB. Trying to shrink with vmware-vdiskmanager -k or vmware-toolbox-cmd disk shrinkonly made no difference.

I have Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize. A more complete dumpe2fs output is available.

Running e4defrag / and e2fsck -E discard didn't allow any more space to be reclaimed (I killed the shrink after the first few vmdks (which started at around 6GB) showed no sign of shrinking.)

What finally did the trick was filling all free space with NULbytes:

dd if=/dev/zero of=wipefile bs=1M; sync; rm wipefile

I could now shrink to a total *.vmdk size of 15.6GB with vmware-toolbox-cmd disk shrinkonly.

That's about 85GB space saving on a VM with 10GB actual data.

It seems that ext4 is not re-using previously used blocks when asked for an unused block, often preferring to give out never-before-used blocks.

Questions

  1. Having old data hanging around longer seems less secure. Why would ext4 not re-use recently-used blocks as soon as possible?

  2. Is there a way to force ext4 to reuse just-used blocks?

  3. Is there a way to prevent a VMware guest's *.vmdk files from continually growing without needing to 0-fill free space on a regular basis?

    • How do you safely (eg not entirely filling the filesystem) automate this?
  • Would you mind to add more technical details, and the reasons you are suspicious? – Rui F Ribeiro Aug 22 '16 at 7:07
  • How now? Anything you can think of that I could check out? – Tom Hale Aug 22 '16 at 7:23
  • Some things are not clear. From the context, are you mentioning the use of ext4 inside the VM? What is the VM doing? What you suspect you can be changed? How are the filesystems created and defined on the VM side? Can reorganising the partitions make a difference? The fs management is highly dependent on the kernel and layer 7 usage, which you have not described. As dding to zero the free space, it should not making a difference except when making tar.gzs of dd. – Rui F Ribeiro Aug 22 '16 at 9:41
  • 1
    I've overhauled the question. The dd definitely was the trick and it's quite a common hack in shrinking linux guest *.vmdk files. The real question is: "why should it be necessary?" – Tom Hale Aug 23 '16 at 4:06
  • 1
    It seems like a limitation with VMWare workstation 12.5.1. Even with scsi0.virtualDev = "pvscsi" in my .vmx, lsscsi -l gives scsi_level=3 which doesn't support UNMAP. – Tom Hale Nov 21 '16 at 12:24
2
  1. Yes, from security standpoint it would be better to immedeately erase any unused blocks. The reason that is not done (ignoring chattr secure-delete flag and patches) is performance. The same is a reson for not using any recenty freed block - it would lead to heavy fragmentations which hurts performance.

  2. No, not really. You could make your whole image much smaller (formatting it to say 15G and then growing only if/when that becomes necessary.) - then it will never grow bigger than 15Gb.

  3. You can try to mount filesystem with discard option - see fstab(5), but I'm not sure if your vmware will heed this

  • As mentioned, e2fsck -E discard didn't do anything, so I doubt that making realtime discards mounting with -o discard will do anything. Also I get: fstrim: /: FITRIM ioctl failed: Operation not supported - trim is not listed as a feature of the virtual SATA device in hdparm -I /dev/dm-0 output. – Tom Hale Aug 23 '16 at 10:45
  • @TomHale did you look at the link? There are changes in vmware config that need to be done too before it could work. But I can't vouch for that part, as said in answer... – Matija Nalis Aug 23 '16 at 10:54
  • I'm currently stuck getting hdparm to correctly see virtual SCSI disks as required by that link - see this question. – Tom Hale Aug 25 '16 at 14:18
  • lsscsi -lL is seeing a virgin SCSI disk as SCSI level 3, so UNMAPwon't work as per your link. I ended up shrinking the filesystem, Logical Volume, Volume Group and Physical Volume... and VMware *.vmdk files are still twice the size the used partition after the shrink. Oh well. – Tom Hale Aug 30 '16 at 9:14
  • @TomHale you should zero-fill all remaining space first to get it smaller - grow PV and VG to maximum size available, create new logical volume with all free space, and then clear it with dd if=/dev/zero of=/dev/mapper/unusedLV bs=1m. Only after that you should lvremove it and shrink VG and PV, and then run vmware image shrinking tools. – Matija Nalis Aug 31 '16 at 8:55
2

No filesystems zero out blocks of deleted files, they just mark those clusters as available. That's why recovery tools can recover deleted files if they weren't overwritten by other files. If filesystem drivers reuse those blocks ASAP then you'll no longer have the ability to recover accidentally erased files and customers will cry, and if they must overwrite the clusters with zero then performance will suffer severely.

Files that are expected to be stored securely should be encrypted instead of leaving as-is on disk. If needed, use shredder tools like shred on Unix and ccleaner drive wiper, eraser, sdelete... on Windows to delete securely.

About VMDK, you should know that it stores sectors in a sparse format, just like VHD, VDI or any other VM's dynamically-sized virtual disk image format. Therefore zeroing out sectors mark them as not needed anymore and the compactor will leave them out, resulting in a smaller file. Any non-zero sectors have to be stored explicitly because the VM doesn't know whether the sector belongs to a deleted file or not

Using dd if=/dev/zero is a bad way because

  • it is slow;
  • it makes the disk image (temporarily) grow to its maximal extent;
  • it (temporarily) uses all free space on the disk, so other concurrent write actions may fail.

as mentioned in the zerofree manpage. Use a tool specifically for zeroing out disks like zerofree instead.

There's no way to prevent a virtual disk image from expanding, because what would you expect if more data need to be written on the disk? Even in case files are deleted, their data are still on disk and take space in the image file. If you don't want the file to grow, the only way is making it fixed-sized on creation.

  • In the case of deleted files, btrfs and ext4 pass down a discard for blocks which are no longer in use. However, Linux distributions check the SCSI version, and unless it is version 5 or greater, it does not send UNMAPs. VMware's driver says SCSI version 3 for me :( – Tom Hale Nov 16 '16 at 5:39
  • I think in this case it's easier to try Virtualbox, which supports almost all common virtual harddisk file types. Or simply write script to zero and compact the vmdk file automatically like what I'm using if you're allowed to shutdown the VM at night – phuclv Nov 22 '16 at 8:11
  • I've found VirtualBox to be buggy in the Hypervisor UI arena, which didn't inspire me to use it for the VM itself. My zero-balloon script is very rough... do you have one that nearly-but-doesn't-quite fill the partitions to avoid disk full conditions? – Tom Hale Nov 23 '16 at 10:16
  • Actually, I'm using LUKS now, so zero-filling would actually make things worse... perhaps using LVM with thin-provisioning with the provisioned size no greater than the actual size is actually the way to go as it processes discards sensibly – Tom Hale Nov 23 '16 at 10:17
0

Possibly because writing a lot of NULs to the VMDK made it reclaim all the deleted blocks and allocate them to wipefile. However, why e4defrag did not do the same is something I do not understand.

  • What do you mean by 'writing to the VMDK'? I'm confused as every filesystem change in the guest is reflected in the VMDK fikes. – Tom Hale Aug 23 '16 at 9:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.