Tag Info

Hot answers tagged

14

You can create a sparse file on certain filesystems, which will appear to be a certain size, but won't actually use that much space on disk. $ dd if=/dev/null of=sparse bs=1024 count=1 seek=524288000 0+0 records in 0+0 records out 0 bytes (0 B) copied, 2.4444e-05 s, 0.0 kB/s $ ls -l sparse -rw-rw-r--. 1 ignacio ignacio 536870912000 May 9 22:25 sparse $ du ...


13

The page you reference (http://intgat.tigress.co.uk/rmy/uml/index.html) states: The utility also works on ext3 or ext4 filesystems. So I'm not sure where you're getting that it doesn't work on ext4 filesystems. Note that the zerofree utility is different from the zerofree kernel patch that is mentioned on the same page (which indeed does not seem to have ...


8

Generally speaking, just use dd, but as you mention the use of KVM virtualization, you might consider using qemu-img: qemu-img create -f raw disk 2G It does the same as the dd command in the answer of Chris Down, effectively. Regardless of what command you use, for use in virtualization, I would strongly suggest to use fallocate to pre-allocate blocks in ...


6

Just for completeness the call for ddrescue: $ ddrescue -S -b8M /dev/sda1 /mount/external/backup/sda1.raw Or with long option: $ ddrescue --sparse --block-size 8M /dev/sda1 /mount/external/backup/sda1.raw Or if you prefer MiBs: $ ddrescue -S -b8Mi /dev/sda1 /mount/external/backup/sda1.raw Note that GNU ddrescue and dd_rescue are different programs. ...


4

There is a similar question on SO. The currently accepted answer by @ephemient suggests using an ioctl called fiemap which is documented in linux/Documentation/filesystems/fiemap.txt. Quoting from that file: The fiemap ioctl is an efficient method for userspace to get file extent mappings. Instead of block-by-block mapping (such as bmap), fiemap ...


4

Ext4 can use 1kB, 2kB or 4kB as the block size; as far as I know the default on Ubuntu is 4kB. Note that here, a block is the size of a file chunk, which is constant for a given filesystem. The file you describe has two blocks that are not zeroes: the one containing hello (surrounded by a bunch of zeroes — 3616 before and 474 after), and the one containing ...


4

bsdtar (at least from libarchive 3.1.2) is able to detect sparse sections using the FS_IOC_FIEMAP ioctl on the file systems that support it (though it supports a number of other APIs as well), however, at least in my test, strangely enough, it is not able to handle the tar files it generates itself (looks like a bug though). However using GNU tar to extract ...


4

cp --sparse=always file-without-holes another-file-with-holes Example: $ cp --sparse=always file-without-holes another-file-with-holes $ du --apparent-size another 16384 another-file-with-holes $ du another-file-with-holes 0 another-file-with-holes


3

I have the same tool installed on Fedora 19, and I noticed in the .spec file a URL which lead to this page titled: Keeping filesystem images sparse. This page included some examples for creating test data so I ran the commands to create the corresponding files. Example $ dd if=/dev/zero of=fs.image bs=1024 seek=2000000 count=0 $ /sbin/mke2fs fs.image $ ls ...


3

The best I could come up with so far is (ksh93, using filefrag from e2fsprogs 1.42.9 (some older versions have a different API), on extent based file systems on Linux): #! /bin/ksh93 export LC_ALL=C for file do filefrag -vb1 -- "$file" | while IFS=": ." read -A a; do [[ $a = +([0-9]) ]] && [[ ${a[@]} != *unwritten* ]] && command ...


3

Sounds like the device is remote. Assuming linux... ssh remote_host 'dd if=/dev/sdb1' | cp --sparse=always /proc/self/fd/0 new-sparse-file If local... dd if=/dev/sdb1 | cp --sparse=always /proc/self/fd/0 new-sparse-file This gives you an image that is mountable. However, if you pulled it across the network then you had 1.2 TB of network traffic ...


3

Linux has a FALLOC_FL_PUNCH_HOLE option to fallocate that can do this. I found a script on github with an example: Using FALLOC_FL_PUNCH_HOLE from Python I modified it a bit to do what you asked -- punch holes in regions of files that are filled with zeros. Here it is: Using FALLOC_FL_PUNCH_HOLE from Python to punch holes in files usage: punch.py [-h] ...


2

You can install the iSCSI Enterprise Target software and setup an iSCSI LUN from the sparse-file like so: In /etc/iet/ietd.conf: Target iqn.my.iscsi.test:disk1 Lun 0 Path=/path/to/my/sparse_file,Type=fileio Then initiate the target from the host you need to restore on. Since the target will show as a local device (eg. /dev/sdd), you can dd from ...


2

You should be able to recover space by using virtio-scsi devices and specifying discard=unmap in the libvirt definition for the disk. I haven't tried this in CentOS/RHEL but I did get this to work in later versions of Fedora. I wrote a blog post about it. I would say to give it a try and see if it works.


1

The "slackspace" you're referring to is often called sparse space within a file. You can use the ls & du commands to detect how much sparse space a give file is consuming. excerpt from Wikipedia's Sparse File Detection The -s option of the ls command shows the occupied space in blocks, and -k the apparent size in blocks too: $ ls -lks ...


1

One word: Journaling. http://www.thegeekstuff.com/2011/05/ext2-ext3-ext4/ As you talk about embedded im assuming you have some form of flash memory? Performance is very spiky on the journaled ext4 on flash. Ext2 is recommended. Here is a good article on disabling journaling and tweaking the fs for no journaling if you must use ext4: ...


1

If you want to make a file sparse you can do that directly with dd. dd if=./zeropadded.iso of=./isnowsparse.iso conv=sparse From the dd(1) manual: sparse If one or more output blocks would consist solely of NUL bytes, try to seek the output file by the required space instead of filling them with NULs, ...


1

There was a patch offered in 2007 to provide sparse file support in GNU dd, but it looks to have not made it into coreutils (at least not as of 8.4). I doubt dd has changed too much since then, the patch might apply against the current version without a lot of work. I'm also really impressed by the creative use of cp in your question, and it got me on the ...


1

If you do not need to seek back in the file, you could use a named pipe: mkfifo my_special_file yes | head -n 100 > my_special_file& cat my_special_file Of course, you can use any other command, including a complex script, to generate data in the pipe.


1

Beyond the portable dd/seek based solution already mentioned some Unixes have specialized commands: At least on Solaris, MacOS/X and Irix: mkfile -n 500m sparseFile On HP-UX prealloc, on AIX lmktemp. and many Linux distributions have truncate


1

dd can create a sparse file, but you would have to manually tell it to copy every extent of used sectors in the filesystem, and you aren't going to do that. What you are looking for is either Ghost4Linux, or clonezilla, both of which can be found on the Parted Magic live cd. They are smart enough to create an image file that only contains the used data, ...


1

The kernel caches the writes and lazily flushes them to disk in the background, allocating disk space as it does so in such a way that it minimizes fragmentation. In other words, you're over thinking things -- don't worry about it. More specifically when it does to to flush some dirty cache buffers, ext4 goes to allocate enough disk space to hold all of ...


1

--size-only tells rsync to skip files which match in size. Since sparse files are preallocated they can be the same size while having different content. I would suggest that you remove the --size-only option and make sure that modification times are synced (--times). rsync then won't check the contents of the file if the modification times match (which is ...


1

One way to go is to create a second disk image, add it to your guest OS and copy files from one to the other. Make the second disk bootable and remove the first one. Another way is to resize your filesystem with resize2fs to its minimum size possible then resize the partition with parted resize to the same or a bit larger size create a new partition in ...


1

It depends on the file system. I don't believe their is a call, which may be why many tools don't handle copying sparse files well. The GNU tool chain use searching for large blocks of zeros as that allows them to remove unused allocated blocks. Many copy tools will convert a sparse file into a file with all blocks allocated. You will likely have to open ...



Only top voted, non community-wiki answers of a minimum length are eligible