Tag Info

Hot answers tagged

17

The page you reference (http://intgat.tigress.co.uk/rmy/uml/index.html) states: The utility also works on ext3 or ext4 filesystems. So I'm not sure where you're getting that it doesn't work on ext4 filesystems. Note that the zerofree utility is different from the zerofree kernel patch that is mentioned on the same page (which indeed does not seem to have ...


15

You can create a sparse file on certain filesystems, which will appear to be a certain size, but won't actually use that much space on disk. $ dd if=/dev/null of=sparse bs=1024 count=1 seek=524288000 0+0 records in 0+0 records out 0 bytes (0 B) copied, 2.4444e-05 s, 0.0 kB/s $ ls -l sparse -rw-rw-r--. 1 ignacio ignacio 536870912000 May 9 22:25 sparse $ du ...


14

You can create a sparse file like this with dd: dd of=file bs=1 seek=2G count=0 $ du file 0 disk $ du --apparent-size file 2097152 disk


13

You want dd_rescue. dd_rescue -a -b 8M /dev/sda1 /mount/external/backup/sda1.raw


9

Edit 2015 as of util-linux 2.25, the fallocate utility on Linux has a -d/--dig-hole option for that. fallocate -d the-file Would dig a hole for every block full of zeros in the file On older systems, you can do it by hand: Linux has a FALLOC_FL_PUNCH_HOLE option to fallocate that can do this. I found a script on github with an example: Using ...


8

Generally speaking, just use dd, but as you mention the use of KVM virtualization, you might consider using qemu-img: qemu-img create -f raw disk 2G It does the same as the dd command in the answer of Chris Down, effectively. Regardless of what command you use, for use in virtualization, I would strongly suggest to use fallocate to pre-allocate blocks in ...


7

Just for completeness the call for ddrescue: $ ddrescue -S -b8M /dev/sda1 /mount/external/backup/sda1.raw Or with long option: $ ddrescue --sparse --block-size 8M /dev/sda1 /mount/external/backup/sda1.raw Or if you prefer MiBs: $ ddrescue -S -b8Mi /dev/sda1 /mount/external/backup/sda1.raw Note that GNU ddrescue and dd_rescue are different programs. ...


7

I have the same tool installed on Fedora 19, and I noticed in the .spec file a URL which lead to this page titled: Keeping filesystem images sparse. This page included some examples for creating test data so I ran the commands to create the corresponding files. Example $ dd if=/dev/zero of=fs.image bs=1024 seek=2000000 count=0 $ /sbin/mke2fs fs.image $ ls ...


7

On the face of it, it's a simple dd: dd if=sparsefile of=sparsefile conv=notrunc bs=1M That reads the entire file, and writes the entire contents back to it. In order to only write the hole itself, you first have to determine where those holes are. You can do that using either filefrag or hdparm: filefrag: # filefrag -e sparsefile Filesystem type is: ...


6

The best I could come up with so far is (ksh93, using filefrag from e2fsprogs 1.42.9 (some older versions have a different API), on extent based file systems on Linux): #! /bin/ksh93 export LC_ALL=C for file do filefrag -vb1 -- "$file" | while IFS=": ." read -A a; do [[ $a = +([0-9]) ]] && [[ ${a[@]} != *unwritten* ]] && command ...


6

bsdtar (at least from libarchive 3.1.2) is able to detect sparse sections using the FS_IOC_FIEMAP ioctl on the file systems that support it (though it supports a number of other APIs as well), however, at least in my test, strangely enough, it is not able to handle the tar files it generates itself (looks like a bug though). However using GNU tar to extract ...


5

See also the GNU truncate command: truncate -s 2G some-file


5

There is a similar question on SO. The currently accepted answer by @ephemient suggests using an ioctl called fiemap which is documented in linux/Documentation/filesystems/fiemap.txt. Quoting from that file: The fiemap ioctl is an efficient method for userspace to get file extent mappings. Instead of block-by-block mapping (such as bmap), fiemap ...


4

Ext4 can use 1kB, 2kB or 4kB as the block size; as far as I know the default on Ubuntu is 4kB. Note that here, a block is the size of a file chunk, which is constant for a given filesystem. The file you describe has two blocks that are not zeroes: the one containing hello (surrounded by a bunch of zeroes — 3616 before and 474 after), and the one containing ...


4

Sounds like the device is remote. Assuming linux... ssh remote_host 'dd if=/dev/sdb1' | cp --sparse=always /proc/self/fd/0 new-sparse-file If local... dd if=/dev/sdb1 | cp --sparse=always /proc/self/fd/0 new-sparse-file This gives you an image that is mountable. However, if you pulled it across the network then you had 1.2 TB of network traffic ...


4

cp --sparse=always file-without-holes another-file-with-holes Example: $ cp --sparse=always file-without-holes another-file-with-holes $ du --apparent-size another 16384 another-file-with-holes $ du another-file-with-holes 0 another-file-with-holes


3

If you want to make a file sparse you can do that directly with dd. dd if=./zeropadded.iso of=./isnowsparse.iso conv=sparse From the dd(1) manual: sparse If one or more output blocks would consist solely of NUL bytes, try to seek the output file by the required space instead of filling them with NULs, ...


3

I haven't tested it, but there is a write-devices patch to rsync, which would solve your problem. You can find the patch in the rsync-patches repository.


2

There was a patch offered in 2007 to provide sparse file support in GNU dd, but it looks to have not made it into coreutils (at least not as of 8.4). I doubt dd has changed too much since then, the patch might apply against the current version without a lot of work. I'm also really impressed by the creative use of cp in your question, and it got me on the ...


2

The kernel caches the writes and lazily flushes them to disk in the background, allocating disk space as it does so in such a way that it minimizes fragmentation. In other words, you're over thinking things -- don't worry about it. More specifically when it does to to flush some dirty cache buffers, ext4 goes to allocate enough disk space to hold all of ...


2

You can install the iSCSI Enterprise Target software and setup an iSCSI LUN from the sparse-file like so: In /etc/iet/ietd.conf: Target iqn.my.iscsi.test:disk1 Lun 0 Path=/path/to/my/sparse_file,Type=fileio Then initiate the target from the host you need to restore on. Since the target will show as a local device (eg. /dev/sdd), you can dd from ...


2

You should be able to recover space by using virtio-scsi devices and specifying discard=unmap in the libvirt definition for the disk. I haven't tried this in CentOS/RHEL but I did get this to work in later versions of Fedora. I wrote a blog post about it. I would say to give it a try and see if it works.


2

Ktorrent is using sparse files. Basically, a sparse file is a file with "holes" : no actual disk space is used for areas of the file which are "empty" (ie contains only zeroes). Your file thus appears to be of a certain size (in your case, the size of your Star Trek video), but is actually not using disk space for those areas that are not yet "filled" (in ...


2

Looks like my manpage of fallocate(1) command line tool is out of date. There is the appropriate option --punch-hole (-p): fallocate -p -o 10G -l 1G qqq


1

One word: Journaling. http://www.thegeekstuff.com/2011/05/ext2-ext3-ext4/ As you talk about embedded im assuming you have some form of flash memory? Performance is very spiky on the journaled ext4 on flash. Ext2 is recommended. Here is a good article on disabling journaling and tweaking the fs for no journaling if you must use ext4: ...


1

The solution is apparently to use cp --sparse=always. My first attempt was with writing some Python code, but unfortunately the MD5 sums didn't match (could anyone tell me why? the code's in edit history).


1

Short of tar-ing it up with a -S flag (assuming GNU tar), and re-executing the scp... no. No utility I'm aware of would have a way of knowing where the "holes" were.


1

If you do not need to seek back in the file, you could use a named pipe: mkfifo my_special_file yes | head -n 100 > my_special_file& cat my_special_file Of course, you can use any other command, including a complex script, to generate data in the pipe.


1

Beyond the portable dd/seek based solution already mentioned some Unixes have specialized commands: At least on Solaris, MacOS/X and Irix: mkfile -n 500m sparseFile On HP-UX prealloc, on AIX lmktemp. and many Linux distributions have truncate


1

dd can create a sparse file, but you would have to manually tell it to copy every extent of used sectors in the filesystem, and you aren't going to do that. What you are looking for is either Ghost4Linux, or clonezilla, both of which can be found on the Parted Magic live cd. They are smart enough to create an image file that only contains the used data, ...



Only top voted, non community-wiki answers of a minimum length are eligible