Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

How to clear unused space with zeros ? (I am asking for tool for btrfs filesystem)

I'm looking for something smarter than

cat /dev/zero > /mnt/X/big_zero ; sync; rm /mnt/X/big_zero

Like FSArchiver is looking for "used space" and ignores unused, but opposite site.

Purpose: I'd like to compress partition images, so filling unused space with zeros is highly recommended.

Btw. For ext3,ext4 : Clear unused space with zeros (ext3,ext4) . Here I ask for btrfs

Clarification of "smarter algorithm": As I see that "smarter algorithm" leads to confusion, here I finally decided to add line about it. Dear reader, please think about filesystem as datastructure. For simplification purposes, let's imagine self balancing red-black tree holding map, that is used as association key->value array, where values are integers. Goal is to clear all negative values with zeros. "not smart" algorithm will iterate over it and add key:value pairs to new map (with eventual altering negative to zero). Smart algorithm will just detect places where negative values are stored and clear them in-place.

share|improve this question
1  
Did you try the sfill suggestion? It looks like it doesn't require extX. –  Kevin Aug 3 '12 at 23:53
    
1. I know you can fill existing files with zeros with shred, but this probably isn't it, right? 2.How are you making those images? Maybe you can do the truncation there? –  Zlatko Aug 4 '12 at 0:23
    
@kevin: that's why i posted the sfill answer in the original question. i rarely use ext2,3,4 filesystems myself so i find utilities that work only on ext* to be annoyingly useless. –  cas Aug 4 '12 at 3:00
    
I need something "smart". Something what only fills empty space with zeros without touching other places - I mean without affecting internal data-structures, like B-Trees (If your are curious, I've written in question "compression" but I need it also for other purposes, like fs images de-duplication, analysis and research) –  Grzegorz Wierzowiecki Aug 5 '12 at 21:57

1 Answer 1

Here's a modified and much expanded repost of my answer to your previous question:

sfill from the secure-delete package can do what you want.

Unlike zerofree (which works only with ext2, ext3, and ext4 filesystems), sfill will work with any file system.

e.g.

sfill -l -l -z /mnt/X

The sfill/secure-delete home page is now seems to have vanished, but it is packaged for debian and ubuntu. probably other distros too. if you need source code, that can be found in the debian archives if you can't find it anywhere else.

NOTE: sfill only works on a mounted filesystem. If you're zero-filling the image files from the host system rather than from within a VM, you'll need to mount the fs image on the host. The exact method varies depending on what kind of image file it is (e.g. qcow2 or raw).

Mounting 'raw' images is easy and straight-forward. Just use kpartx on the image file to create a loopback device and /dev/mapper/ entries for each partition can then be mounted individually.

here's a partial script modified from a fragment of the script I use to mount net-bootable freedos hard-disk images (used for bios updates on machines where flashrom won't work):

image="myrawimagefile.img"

# use kpartx to build /dev/mapper device nodes
KP=$(kpartx -a -v "$image")

# now mount each partition under ./hd/
for p in $(echo "$KP" | awk '/^add map/ {print $3}') ; do
  dm="/dev/mapper/$p"
  fp=$(echo "$p" | sed -r -e 's/^loop[0-9]+//')
  mkdir -p "./hd/$fp"
  mount "$dm" "./hd/$fp"
  sfill -l -l -v "./hd/$fp"
  umount "$dm"
done

# now remove the loopback device and /dev/mapper entries
kpartx -d "$image"

NOTE: it assumes every partition in the image is mountable. true in my use-case (a freedos hd image with one partition). not true if one of the partitions is, say, swap space. Detecting such partitions and using dd to zero the swap partition is left as an exercise for the reader :)

This method, or a variant of it, should work for LVM volumes too.

If your images are qcow then you can use the qemu-nbd tool from the qemu-utils package, which will present the image file and its partitions as network block devices - e.g. /dev/nbd0 /dev/ndb0p1 - which can be used in a similar fashion to the /dev/mapper devices above.

It's possibly easier and less hassle (but more time-consuming) to use qemu-img to convert from qcow to raw, use the method for raw above, and then convert the modified raw image back to compressed qcow2. This will probably result in slightly smaller images than using qemu-nbd as you'll be compressing the new qcow2 image as it is being created.

share|improve this answer
    
adding error checking would also be a good idea. in particular, don't run sfill -l -l -v "./hd/$fp" if the previous mount failed. that's why i called it only a partial script - it's not complete or ready for use as-is. –  cas Aug 4 '12 at 3:42
    
Could you ensure me if this tool fills free space, until possible, and then deletes file ? (in nutshell, I know it's for "wiping" and supports multi-passes, I want to ensure about basic idea of clearing with zeros) –  Grzegorz Wierzowiecki Aug 4 '12 at 19:27
    
To the best of my knowledge, that's what it does. the -l -l make it just zero the empty space without the multiple passes of random data. For quick testing, I suggest making a small (say, 100MB) disk image, mounting it, and running sfill -l -l on it. –  cas Aug 5 '12 at 10:00
1  
Please describe a "smart" algorithm that isn't specific to a particular filesystem family (like zerofill is for ext2/3/4). You wanted a program that could do the job. You didn't say it had to be magical as well. –  cas Aug 5 '12 at 13:08
1  
and, really, if the filesystem is not in use (because, e.g., it's mounted on the host for zerofilling) what does it matter how smart the algorithm is? Trawling through the fs metadata looking for empty sectors isn't going to be any faster or better than just writing a huge fill full of NULs...the filesystem itself is already plenty smart enough for figuring out where empty space is. –  cas Aug 5 '12 at 13:13

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.