Hot answers tagged

17

The page you reference (http://intgat.tigress.co.uk/rmy/uml/index.html) states: The utility also works on ext3 or ext4 filesystems. So I'm not sure where you're getting that it doesn't work on ext4 filesystems. Note that the zerofree utility is different from the zerofree kernel patch that is mentioned on the same page (which indeed does not seem to have ...


16

You can create a sparse file on certain filesystems, which will appear to be a certain size, but won't actually use that much space on disk. $ dd if=/dev/null of=sparse bs=1024 count=1 seek=524288000 0+0 records in 0+0 records out 0 bytes (0 B) copied, 2.4444e-05 s, 0.0 kB/s $ ls -l sparse -rw-rw-r--. 1 ignacio ignacio 536870912000 May 9 22:25 sparse $ du ...


14

You want dd_rescue. dd_rescue -a -b 8M /dev/sda1 /mount/external/backup/sda1.raw


14

You can create a sparse file like this with dd: dd of=file bs=1 seek=2G count=0 $ du file 0 disk $ du --apparent-size file 2097152 disk


14

Edit 2015 as of util-linux 2.25, the fallocate utility on Linux has a -d/--dig-hole option for that. fallocate -d the-file Would dig a hole for every block full of zeros in the file On older systems, you can do it by hand: Linux has a FALLOC_FL_PUNCH_HOLE option to fallocate that can do this. I found a script on github with an example: Using ...


9

On the face of it, it's a simple dd: dd if=sparsefile of=sparsefile conv=notrunc bs=1M That reads the entire file, and writes the entire contents back to it. In order to only write the hole itself, you first have to determine where those holes are. You can do that using either filefrag or hdparm: filefrag: # filefrag -e sparsefile Filesystem type is: ...


8

I have the same tool installed on Fedora 19, and I noticed in the .spec file a URL which lead to this page titled: Keeping filesystem images sparse. This page included some examples for creating test data so I ran the commands to create the corresponding files. Example $ dd if=/dev/zero of=fs.image bs=1024 seek=2000000 count=0 $ /sbin/mke2fs fs.image $ ls ...


8

Generally speaking, just use dd, but as you mention the use of KVM virtualization, you might consider using qemu-img: qemu-img create -f raw disk 2G It does the same as the dd command in the answer of Chris Down, effectively. Regardless of what command you use, for use in virtualization, I would strongly suggest to use fallocate to pre-allocate blocks in ...


8

Just for completeness the call for ddrescue: $ ddrescue -S -b8M /dev/sda1 /mount/external/backup/sda1.raw Or with long option: $ ddrescue --sparse --block-size 8M /dev/sda1 /mount/external/backup/sda1.raw Or if you prefer MiBs: $ ddrescue -S -b8Mi /dev/sda1 /mount/external/backup/sda1.raw Note that GNU ddrescue and dd_rescue are different programs. ...


7

bsdtar (at least from libarchive 3.1.2) is able to detect sparse sections using the FS_IOC_FIEMAP ioctl on the file systems that support it (though it supports a number of other APIs as well), however, at least in my test, strangely enough, it is not able to handle the tar files it generates itself (looks like a bug though). However using GNU tar to extract ...


6

The best I could come up with so far is (ksh93, using filefrag from e2fsprogs 1.42.9 (some older versions have a different API), on extent based file systems on Linux): #! /bin/ksh93 export LC_ALL=C for file do filefrag -vb1 -- "$file" | while IFS=": ." read -A a; do [[ $a = +([0-9]) ]] && [[ ${a[@]} != *unwritten* ]] && command ...


6

See also the GNU truncate command: truncate -s 2G some-file


5

There is a similar question on SO. The currently accepted answer by @ephemient suggests using an ioctl called fiemap which is documented in linux/Documentation/filesystems/fiemap.txt. Quoting from that file: The fiemap ioctl is an efficient method for userspace to get file extent mappings. Instead of block-by-block mapping (such as bmap), fiemap ...


4

Sounds like the device is remote. Assuming linux... ssh remote_host 'dd if=/dev/sdb1' | cp --sparse=always /proc/self/fd/0 new-sparse-file If local... dd if=/dev/sdb1 | cp --sparse=always /proc/self/fd/0 new-sparse-file This gives you an image that is mountable. However, if you pulled it across the network then you had 1.2 TB of network traffic ...


4

Ext4 can use 1kB, 2kB or 4kB as the block size; as far as I know the default on Ubuntu is 4kB. Note that here, a block is the size of a file chunk, which is constant for a given filesystem. The file you describe has two blocks that are not zeroes: the one containing hello (surrounded by a bunch of zeroes — 3616 before and 474 after), and the one containing ...


4

cp --sparse=always file-without-holes another-file-with-holes Example: $ cp --sparse=always file-without-holes another-file-with-holes $ du --apparent-size another 16384 another-file-with-holes $ du another-file-with-holes 0 another-file-with-holes


3

If you want to make a file sparse you can do that directly with dd. dd if=./zeropadded.iso of=./isnowsparse.iso conv=sparse From the dd(1) manual: sparse If one or more output blocks would consist solely of NUL bytes, try to seek the output file by the required space instead of filling them with NULs, ...


3

Looks like my manpage of fallocate(1) command line tool is out of date. There is the appropriate option --punch-hole (-p): fallocate -p -o 10G -l 1G qqq


3

I haven't tested it, but there is a write-devices patch to rsync, which would solve your problem. You can find the patch in the rsync-patches repository.


2

You can install the iSCSI Enterprise Target software and setup an iSCSI LUN from the sparse-file like so: In /etc/iet/ietd.conf: Target iqn.my.iscsi.test:disk1 Lun 0 Path=/path/to/my/sparse_file,Type=fileio Then initiate the target from the host you need to restore on. Since the target will show as a local device (eg. /dev/sdd), you can dd from ...


2

The kernel caches the writes and lazily flushes them to disk in the background, allocating disk space as it does so in such a way that it minimizes fragmentation. In other words, you're over thinking things -- don't worry about it. More specifically when it does to to flush some dirty cache buffers, ext4 goes to allocate enough disk space to hold all of ...


2

One way to go is to create a second disk image, add it to your guest OS and copy files from one to the other. Make the second disk bootable and remove the first one. Another way is to resize your filesystem with resize2fs to its minimum size possible then resize the partition with parted resize to the same or a bit larger size create a new partition in ...


2

There was a patch offered in 2007 to provide sparse file support in GNU dd, but it looks to have not made it into coreutils (at least not as of 8.4). I doubt dd has changed too much since then, the patch might apply against the current version without a lot of work. I'm also really impressed by the creative use of cp in your question, and it got me on the ...


2

Ktorrent is using sparse files. Basically, a sparse file is a file with "holes" : no actual disk space is used for areas of the file which are "empty" (ie contains only zeroes). Your file thus appears to be of a certain size (in your case, the size of your Star Trek video), but is actually not using disk space for those areas that are not yet "filled" (in ...


2

You should be able to recover space by using virtio-scsi devices and specifying discard=unmap in the libvirt definition for the disk. I haven't tried this in CentOS/RHEL but I did get this to work in later versions of Fedora. I wrote a blog post about it. I would say to give it a try and see if it works.


2

This article has some useful suggestions for rsync at least: Problems Using rsync --sparse works, but causes a huge a mount of unnecessary disk writes. Changing 10 bytes on 50GB long (1GB used) should cause only one or two blocks to be written, this causes 1GB to be written. This is slow, and possible not good for the disks' longevity. Using ...


1

The solution is apparently to use cp --sparse=always. My first attempt was with writing some Python code, but unfortunately the MD5 sums didn't match (could anyone tell me why? the code's in edit history).


1

One word: Journaling. http://www.thegeekstuff.com/2011/05/ext2-ext3-ext4/ As you talk about embedded im assuming you have some form of flash memory? Performance is very spiky on the journaled ext4 on flash. Ext2 is recommended. Here is a good article on disabling journaling and tweaking the fs for no journaling if you must use ext4: ...


1

file.vdi is in all likelihood a sparse file. This is very common with virtual machine disk images: parts that have never been written to are left as holes in the file that don't consume space. You can confirm by checking whether the length of the original file matches its disk usage: ls -l file.vdi; du file.dvi I expect that ls -l will report 14GB (actual ...


1

Do both use the same filesystem? Different filesystems may have different blocksizes so the filesize may be different.



Only top voted, non community-wiki answers of a minimum length are eligible