Tag Info

Hot answers tagged

15

The page you reference (http://intgat.tigress.co.uk/rmy/uml/index.html) states: The utility also works on ext3 or ext4 filesystems. So I'm not sure where you're getting that it doesn't work on ext4 filesystems. Note that the zerofree utility is different from the zerofree kernel patch that is mentioned on the same page (which indeed does not seem to have ...


14

You can create a sparse file like this with dd: dd of=file bs=1 seek=2G count=0 $ du file 0 disk $ du --apparent-size file 2097152 disk


14

You can create a sparse file on certain filesystems, which will appear to be a certain size, but won't actually use that much space on disk. $ dd if=/dev/null of=sparse bs=1024 count=1 seek=524288000 0+0 records in 0+0 records out 0 bytes (0 B) copied, 2.4444e-05 s, 0.0 kB/s $ ls -l sparse -rw-rw-r--. 1 ignacio ignacio 536870912000 May 9 22:25 sparse $ du ...


12

You want dd_rescue. dd_rescue -a -b 8M /dev/sda1 /mount/external/backup/sda1.raw


8

Generally speaking, just use dd, but as you mention the use of KVM virtualization, you might consider using qemu-img: qemu-img create -f raw disk 2G It does the same as the dd command in the answer of Chris Down, effectively. Regardless of what command you use, for use in virtualization, I would strongly suggest to use fallocate to pre-allocate blocks in ...


7

I have the same tool installed on Fedora 19, and I noticed in the .spec file a URL which lead to this page titled: Keeping filesystem images sparse. This page included some examples for creating test data so I ran the commands to create the corresponding files. Example $ dd if=/dev/zero of=fs.image bs=1024 seek=2000000 count=0 $ /sbin/mke2fs fs.image $ ls ...


7

Just for completeness the call for ddrescue: $ ddrescue -S -b8M /dev/sda1 /mount/external/backup/sda1.raw Or with long option: $ ddrescue --sparse --block-size 8M /dev/sda1 /mount/external/backup/sda1.raw Or if you prefer MiBs: $ ddrescue -S -b8Mi /dev/sda1 /mount/external/backup/sda1.raw Note that GNU ddrescue and dd_rescue are different programs. ...


6

Linux has a FALLOC_FL_PUNCH_HOLE option to fallocate that can do this. I found a script on github with an example: Using FALLOC_FL_PUNCH_HOLE from Python I modified it a bit to do what you asked -- punch holes in regions of files that are filled with zeros. Here it is: Using FALLOC_FL_PUNCH_HOLE from Python to punch holes in files usage: punch.py [-h] ...


6

The best I could come up with so far is (ksh93, using filefrag from e2fsprogs 1.42.9 (some older versions have a different API), on extent based file systems on Linux): #! /bin/ksh93 export LC_ALL=C for file do filefrag -vb1 -- "$file" | while IFS=": ." read -A a; do [[ $a = +([0-9]) ]] && [[ ${a[@]} != *unwritten* ]] && command ...


5

See also the GNU truncate command: truncate -s 2G some-file


4

bsdtar (at least from libarchive 3.1.2) is able to detect sparse sections using the FS_IOC_FIEMAP ioctl on the file systems that support it (though it supports a number of other APIs as well), however, at least in my test, strangely enough, it is not able to handle the tar files it generates itself (looks like a bug though). However using GNU tar to extract ...


4

cp --sparse=always file-without-holes another-file-with-holes Example: $ cp --sparse=always file-without-holes another-file-with-holes $ du --apparent-size another 16384 another-file-with-holes $ du another-file-with-holes 0 another-file-with-holes


4

There is a similar question on SO. The currently accepted answer by @ephemient suggests using an ioctl called fiemap which is documented in linux/Documentation/filesystems/fiemap.txt. Quoting from that file: The fiemap ioctl is an efficient method for userspace to get file extent mappings. Instead of block-by-block mapping (such as bmap), fiemap ...


4

Ext4 can use 1kB, 2kB or 4kB as the block size; as far as I know the default on Ubuntu is 4kB. Note that here, a block is the size of a file chunk, which is constant for a given filesystem. The file you describe has two blocks that are not zeroes: the one containing hello (surrounded by a bunch of zeroes — 3616 before and 474 after), and the one containing ...


4

Sounds like the device is remote. Assuming linux... ssh remote_host 'dd if=/dev/sdb1' | cp --sparse=always /proc/self/fd/0 new-sparse-file If local... dd if=/dev/sdb1 | cp --sparse=always /proc/self/fd/0 new-sparse-file This gives you an image that is mountable. However, if you pulled it across the network then you had 1.2 TB of network traffic ...


3

I haven't tested it, but there is a write-devices patch to rsync, which would solve your problem. You can find the patch in the rsync-patches repository.


2

There was a patch offered in 2007 to provide sparse file support in GNU dd, but it looks to have not made it into coreutils (at least not as of 8.4). I doubt dd has changed too much since then, the patch might apply against the current version without a lot of work. I'm also really impressed by the creative use of cp in your question, and it got me on the ...


2

You can install the iSCSI Enterprise Target software and setup an iSCSI LUN from the sparse-file like so: In /etc/iet/ietd.conf: Target iqn.my.iscsi.test:disk1 Lun 0 Path=/path/to/my/sparse_file,Type=fileio Then initiate the target from the host you need to restore on. Since the target will show as a local device (eg. /dev/sdd), you can dd from ...


2

You should be able to recover space by using virtio-scsi devices and specifying discard=unmap in the libvirt definition for the disk. I haven't tried this in CentOS/RHEL but I did get this to work in later versions of Fedora. I wrote a blog post about it. I would say to give it a try and see if it works.


1

One word: Journaling. http://www.thegeekstuff.com/2011/05/ext2-ext3-ext4/ As you talk about embedded im assuming you have some form of flash memory? Performance is very spiky on the journaled ext4 on flash. Ext2 is recommended. Here is a good article on disabling journaling and tweaking the fs for no journaling if you must use ext4: ...


1

The solution is apparently to use cp --sparse=always. My first attempt was with writing some Python code, but unfortunately the MD5 sums didn't match (could anyone tell me why? the code's in edit history).


1

If you want to make a file sparse you can do that directly with dd. dd if=./zeropadded.iso of=./isnowsparse.iso conv=sparse From the dd(1) manual: sparse If one or more output blocks would consist solely of NUL bytes, try to seek the output file by the required space instead of filling them with NULs, ...


1

Short of tar-ing it up with a -S flag (assuming GNU tar), and re-executing the scp... no. No utility I'm aware of would have a way of knowing where the "holes" were.


1

If you do not need to seek back in the file, you could use a named pipe: mkfifo my_special_file yes | head -n 100 > my_special_file& cat my_special_file Of course, you can use any other command, including a complex script, to generate data in the pipe.


1

Beyond the portable dd/seek based solution already mentioned some Unixes have specialized commands: At least on Solaris, MacOS/X and Irix: mkfile -n 500m sparseFile On HP-UX prealloc, on AIX lmktemp. and many Linux distributions have truncate


1

dd can create a sparse file, but you would have to manually tell it to copy every extent of used sectors in the filesystem, and you aren't going to do that. What you are looking for is either Ghost4Linux, or clonezilla, both of which can be found on the Parted Magic live cd. They are smart enough to create an image file that only contains the used data, ...


1

The kernel caches the writes and lazily flushes them to disk in the background, allocating disk space as it does so in such a way that it minimizes fragmentation. In other words, you're over thinking things -- don't worry about it. More specifically when it does to to flush some dirty cache buffers, ext4 goes to allocate enough disk space to hold all of ...


1

--size-only tells rsync to skip files which match in size. Since sparse files are preallocated they can be the same size while having different content. I would suggest that you remove the --size-only option and make sure that modification times are synced (--times). rsync then won't check the contents of the file if the modification times match (which is ...


1

One way to go is to create a second disk image, add it to your guest OS and copy files from one to the other. Make the second disk bootable and remove the first one. Another way is to resize your filesystem with resize2fs to its minimum size possible then resize the partition with parted resize to the same or a bit larger size create a new partition in ...



Only top voted, non community-wiki answers of a minimum length are eligible