Hot answers tagged

17

The page you reference (http://intgat.tigress.co.uk/rmy/uml/index.html) states: The utility also works on ext3 or ext4 filesystems. So I'm not sure where you're getting that it doesn't work on ext4 filesystems. Note that the zerofree utility is different from the zerofree kernel patch that is mentioned on the same page (which indeed does not seem to have ...


17

Edit 2015 as of util-linux 2.25, the fallocate utility on Linux has a -d/--dig-hole option for that. fallocate -d the-file Would dig a hole for every block full of zeros in the file On older systems, you can do it by hand: Linux has a FALLOC_FL_PUNCH_HOLE option to fallocate that can do this. I found a script on github with an example: Using ...


16

You can create a sparse file on certain filesystems, which will appear to be a certain size, but won't actually use that much space on disk. $ dd if=/dev/null of=sparse bs=1024 count=1 seek=524288000 0+0 records in 0+0 records out 0 bytes (0 B) copied, 2.4444e-05 s, 0.0 kB/s $ ls -l sparse -rw-rw-r--. 1 ignacio ignacio 536870912000 May 9 22:25 sparse $ du ...


14

You want dd_rescue. dd_rescue -a -b 8M /dev/sda1 /mount/external/backup/sda1.raw


14

You can create a sparse file like this with dd: dd of=file bs=1 seek=2G count=0 $ du file 0 disk $ du --apparent-size file 2097152 disk


9

On the face of it, it's a simple dd: dd if=sparsefile of=sparsefile conv=notrunc bs=1M That reads the entire file, and writes the entire contents back to it. In order to only write the hole itself, you first have to determine where those holes are. You can do that using either filefrag or hdparm: filefrag: # filefrag -e sparsefile Filesystem type is: ...


8

Just for completeness the call for ddrescue: $ ddrescue -S -b8M /dev/sda1 /mount/external/backup/sda1.raw Or with long option: $ ddrescue --sparse --block-size 8M /dev/sda1 /mount/external/backup/sda1.raw Or if you prefer MiBs: $ ddrescue -S -b8Mi /dev/sda1 /mount/external/backup/sda1.raw Note that GNU ddrescue and dd_rescue are different programs. ...


8

I have the same tool installed on Fedora 19, and I noticed in the .spec file a URL which lead to this page titled: Keeping filesystem images sparse. This page included some examples for creating test data so I ran the commands to create the corresponding files. Example $ dd if=/dev/zero of=fs.image bs=1024 seek=2000000 count=0 $ /sbin/mke2fs fs.image $ ls ...


8

Generally speaking, just use dd, but as you mention the use of KVM virtualization, you might consider using qemu-img: qemu-img create -f raw disk 2G It does the same as the dd command in the answer of Chris Down, effectively. Regardless of what command you use, for use in virtualization, I would strongly suggest to use fallocate to pre-allocate blocks in ...


7

bsdtar (at least from libarchive 3.1.2) is able to detect sparse sections using the FS_IOC_FIEMAP ioctl on the file systems that support it (though it supports a number of other APIs as well), however, at least in my test, strangely enough, it is not able to handle the tar files it generates itself (looks like a bug though). However using GNU tar to extract ...


6

See also the GNU truncate command: truncate -s 2G some-file


6

The best I could come up with so far is (ksh93, using filefrag from e2fsprogs 1.42.9 (some older versions have a different API), on extent based file systems on Linux): #! /bin/ksh93 export LC_ALL=C for file do filefrag -vb1 -- "$file" | while IFS=": ." read -A a; do [[ $a = +([0-9]) ]] && [[ ${a[@]} != *unwritten* ]] && command ...


5

There is a similar question on SO. The currently accepted answer by @ephemient suggests using an ioctl called fiemap which is documented in linux/Documentation/filesystems/fiemap.txt. Quoting from that file: The fiemap ioctl is an efficient method for userspace to get file extent mappings. Instead of block-by-block mapping (such as bmap), fiemap ...


5

If you want to make a file sparse you can do that directly with dd. dd if=./zeropadded.iso of=./isnowsparse.iso conv=sparse From the dd(1) manual: sparse If one or more output blocks would consist solely of NUL bytes, try to seek the output file by the required space instead of filling them with NULs, ...


4

cp --sparse=always file-without-holes another-file-with-holes Example: $ cp --sparse=always file-without-holes another-file-with-holes $ du --apparent-size another 16384 another-file-with-holes $ du another-file-with-holes 0 another-file-with-holes


4

Ext4 can use 1kB, 2kB or 4kB as the block size; as far as I know the default on Ubuntu is 4kB. Note that here, a block is the size of a file chunk, which is constant for a given filesystem. The file you describe has two blocks that are not zeroes: the one containing hello (surrounded by a bunch of zeroes — 3616 before and 474 after), and the one containing ...


4

Sounds like the device is remote. Assuming linux... ssh remote_host 'dd if=/dev/sdb1' | cp --sparse=always /proc/self/fd/0 new-sparse-file If local... dd if=/dev/sdb1 | cp --sparse=always /proc/self/fd/0 new-sparse-file This gives you an image that is mountable. However, if you pulled it across the network then you had 1.2 TB of network traffic ...


3

I haven't tested it, but there is a write-devices patch to rsync, which would solve your problem. You can find the patch in the rsync-patches repository.


3

Looks like my manpage of fallocate(1) command line tool is out of date. There is the appropriate option --punch-hole (-p): fallocate -p -o 10G -l 1G qqq


3

Some quick answers: first, you didn't create a sparse file. Try these extra commands dd if=/tmp/BIL of=/tmp/sparse seek=1000 ls -ls /tmp/sparse You will see the size is 512003 bytes, but only takes 8 blocks. The null bytes have to occupy a whole block, and be on a block boundary for them to be possibly sparse in the filesystem. Why does the second ...


2

One way to go is to create a second disk image, add it to your guest OS and copy files from one to the other. Make the second disk bootable and remove the first one. Another way is to resize your filesystem with resize2fs to its minimum size possible then resize the partition with parted resize to the same or a bit larger size create a new partition in ...


2

There was a patch offered in 2007 to provide sparse file support in GNU dd, but it looks to have not made it into coreutils (at least not as of 8.4). I doubt dd has changed too much since then, the patch might apply against the current version without a lot of work. I'm also really impressed by the creative use of cp in your question, and it got me on the ...


2

The kernel caches the writes and lazily flushes them to disk in the background, allocating disk space as it does so in such a way that it minimizes fragmentation. In other words, you're over thinking things -- don't worry about it. More specifically when it does to to flush some dirty cache buffers, ext4 goes to allocate enough disk space to hold all of ...


2

Short of tar-ing it up with a -S flag (assuming GNU tar), and re-executing the scp... no. No utility I'm aware of would have a way of knowing where the "holes" were.


2

You can install the iSCSI Enterprise Target software and setup an iSCSI LUN from the sparse-file like so: In /etc/iet/ietd.conf: Target iqn.my.iscsi.test:disk1 Lun 0 Path=/path/to/my/sparse_file,Type=fileio Then initiate the target from the host you need to restore on. Since the target will show as a local device (eg. /dev/sdd), you can dd from ...


2

This article has some useful suggestions for rsync at least: Problems Using rsync --sparse works, but causes a huge a mount of unnecessary disk writes. Changing 10 bytes on 50GB long (1GB used) should cause only one or two blocks to be written, this causes 1GB to be written. This is slow, and possible not good for the disks' longevity. Using ...


2

You should be able to recover space by using virtio-scsi devices and specifying discard=unmap in the libvirt definition for the disk. I haven't tried this in CentOS/RHEL but I did get this to work in later versions of Fedora. I wrote a blog post about it. I would say to give it a try and see if it works.


2

I've had good luck with this: cd whatever rsync -avxWSHAXI . . The -Iforces rsync to update all files, regardless of whether it thinks they've changed or not; the -S causes the new files to be sparsified. -a makes it happen recursively so you can sparsify whole directory trees in one command. It's not as good as a bespoke tool which hunts out holes and ...


2

Ktorrent is using sparse files. Basically, a sparse file is a file with "holes" : no actual disk space is used for areas of the file which are "empty" (ie contains only zeroes). Your file thus appears to be of a certain size (in your case, the size of your Star Trek video), but is actually not using disk space for those areas that are not yet "filled" (in ...


1

One word: Journaling. http://www.thegeekstuff.com/2011/05/ext2-ext3-ext4/ As you talk about embedded im assuming you have some form of flash memory? Performance is very spiky on the journaled ext4 on flash. Ext2 is recommended. Here is a good article on disabling journaling and tweaking the fs for no journaling if you must use ext4: ...



Only top voted, non community-wiki answers of a minimum length are eligible