Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [sparse-files]

The tag has no usage guidance.

1
vote
2answers
33 views

Can losetup be made efficient with sparse files?

So my setup is like this. $ truncate -s 1T volume $ losetup -f --show volume /dev/loop0 $ mkfs.ext4 /dev/loop0 $ ls -sh volume 1.1G volume $ mount /dev/loop0 /mnt/loop Now I have a 1.1TB volume, as ...
0
votes
1answer
33 views

Is it possible to zero the free space on a LUKS volume?

I want to have a LUKS volume as a loopback device pointing at a sparse file. Lets say I put a 20gb file on it, then later I delete it. How can I zero that area on the device itself so I can fallocate -...
0
votes
1answer
185 views

rsync: is possible to copy only the used space of qcow2 or raw image?

Simple question. I have a 200G qcow2 or raw disk image, but of 200G only 30G are used. If I do rsync -avP copy all the 200G on usb disk or network and it wast time,space and network bandwidth. Is ...
1
vote
1answer
614 views

qcow2 actual size

I am little bit confused with the real size of qcow2 files. ls -alh VMs/ubuntu-mini.qcow2 -rw------- 1 root root 21G mar 31 23:15 VMs/ubuntu-mini.qcow2 du -h VMs/ubuntu-mini.qcow2 2,7G VMs/ubuntu-...
4
votes
2answers
942 views

What is the difference between `fallocate --dig-holes` and `fallocate --punch-hole` in Linux?

I totally understand that --dig-holes creates a sparse file in-place. That is, if the file has holes --dig-holes options removes those holes: Let's take it in a very simplified way, let's say we ...
9
votes
1answer
699 views

Are there any disadvantages of `cp --sparse=always`?

Is there any reason not to use use --sparse=always with every invocation to cp? info cp says: ‘--sparse=WHEN’ A “sparse file” contains “holes”—a sequence of zero bytes that does not occupy ...
0
votes
1answer
76 views

Can we allocate file holes to another file?

With reference to the Advanced Programming in the Unix Environment the file holes are created when we try to seek past the end of the file and write something there.For example, int x = lseek(fd,...
2
votes
1answer
692 views

What are functions to manupulate sparse files under Linux?

What are functions to manupulate sparse files under Linux? (let's say in C, notes about other systems highly welcome) e.g.: make hole inside of file by removing part of its inside investigate ...
2
votes
1answer
140 views

Files created by fallocate compared to normal text files

I am testing S3 command line tool performances at the moment and I am testing them using files I have created with fallocate. Im measuring the time different tools take to upload, say, 200 files of ...
12
votes
1answer
325 views

What could explain this strange sparse file handling of/in tmpfs?

On my ext4 filesystem partition I can run the following code: fs="/mnt/ext4" #create sparse 100M file on ${fs} dd if=/dev/zero \ of=${fs}/sparse100M conv=sparse seek=$((100*2*1024-1)) count=1 2&...
6
votes
1answer
2k views

rsync of sparse qemu image increases disk size

I have a sparse raw qemu image that I want to transfer to another server. qemu-img info gives me: image: sparse.img file format: raw virtual size: 50G disk size: 16G I transfer it with: rsync -azhP ...
3
votes
1answer
175 views

Output of ls command confusing for KVM raw image on XFS

After creating a new VM on a raw image using virt-install, I tried to verify that the image was a sparse file, but was unable to determine using the ls command. ls -ls myvm.img 1048576000 -rw------- ...
14
votes
3answers
511 views

How do I output “sparseness” of a file?

How do I output how much of file nominal size is actually filled with data? Like vmtouch shows how much of file is currently in memory... I expect the workflow to be like this: $ fallocate -l ...
0
votes
2answers
114 views

parsing using loop and assigning row numbers to file

It is a bit complicated. i have a configuration file. it looks something like this. customer="airtel" model_name=fast programmer_typ_of="dev" Now the 1st column contains all the keys and whatever is ...
8
votes
2answers
316 views

Why is fragmentation level so huge in files that contain other filesystems?

I've just found out what sparse files are and wanted to conduct some experiments on them. On wiki you can read that the files can get fragmented easily. I wanted to check how bad that is. I created a ...
6
votes
2answers
2k views

Understanding sparse files, dd, seek, inode block structure

At work we use sparse files as part of out Oracle VM environment for the guest disk images. After some questions from a colleague (which have since been answered) I am left with more questions about ...
1
vote
1answer
253 views

Create and use sparse file as non-privileged user

Here is my setup: I am non-privileged user (non-root). I cannot use sudo, nor I can use any other command to switch to root (e.g. su), or run commands as root. As a consequence, I cannot modify the /...
0
votes
2answers
2k views

How to get number of files in a 7z

I want to get the number of files (included directories) contained in the root of a 7z archive, but the 7z command on Linux don’t bring a simple way to only get the number and noting else. For ...
1
vote
1answer
743 views

rsync generating a destination directory of bigger size than source?

We are copying our data from source server to destination server using rsync : rsync -av --progress --sparse user@destination server After rsync, we observed that destination directory is bigger in ...
1
vote
0answers
51 views

xfs: writing to 17TB offset doesn't actually write it to specified offset

I have a situation wherein I want to write a few GBs at a very large offset - say 17TB. But when I use the dd command to do so, I notice that the filesystem is doing some kind of optimization - sparse ...
1
vote
0answers
848 views

Creating a sparse (automatically expanding and contracting) swap file with hole-punching support

I have this crazy idea, and want to know the pitfalls of such an approach (or whether it can even work with no deadlocks.. etc). The goal is to create an automatically expanding and contracting ...
3
votes
2answers
1k views

How do I punch a hole (FALLOC_FL_PUNCH_HOLE) in a file from command line?

There is fallocate(2) system call to make sparse files by erasing a "holes" in them. Is there a command line tool for it? Expecting something like: makehole myfile.dat --offset=10G --length=1G
1
vote
1answer
91 views

file (vdi) increase (at double) size after split and then cat operation

I'd splitted a vdi file with 7 gigabytes and then joined with cat command. The file works normally on VBox, but get 14 gigabytes after be re-joined. The exact operation was: $ split -- bytes=2000M ...
3
votes
1answer
264 views

“Ghost” files in ext4? (ktorrent allocated disk space)

I'm downloading with KTorrent, and have noticed that when I select "allocate disk space before starting torrent", it creates all the files at start, presumably empty, to avoid fragmentation. When I'm ...
2
votes
1answer
497 views

What is the correct syntax for cloning a ntfs partition to a sparse file on another ntfs drive?

I have tried ddrescue -v -v -S /dev/sda2 /media/mint/otherhdd/img /media/mint/otherhdd/log and ddrescue -v -v -d -S /dev/sda2 /media/mint/otherhdd/img /media/mint/otherhdd/log but it does not work ...
3
votes
1answer
674 views

From a sparse file to a block device

I have a system image sparse file which actual size is only a few MBs but the "apparent size" is about 1GB. I'm trying to write it to a block device efficiently (without the holes). Here are some ...
8
votes
1answer
5k views

Converting sparse file to non-sparse in place

On Linux, given a sparse file, how to make it non-sparse, in place? It could be copied with cp --sparse=never ..., but if the file is say 10G and the hole is 2G (that is the allocated space is 8G), ...
4
votes
1answer
517 views

Linux: How to calculate/find the slackspace of a file?

How would I go about finding the slackspace of a file, for instance. fls -o 63 /images/diskimg.dd istat -o 63 /images/diskimg.dd 31 icat -o 63 /images/diskimg.dd | xxd | less In which of these ...
8
votes
1answer
192 views

How to display the non-sparse parts of a sparse file?

Imagine a file created with: truncate -s1T file echo test >> file truncate -s2T file I now have a 2 tebibyte file (that occupies 4kiB on disk), with "test\n" written in the middle. How would ...
9
votes
2answers
5k views

How can I speed up operations on sparse files with tar, gzip, rsync?

I have a sparse file. (du -h reports 3G and du -h --apparent-size reports 100G.) So far, so good. Now, when I want to compress the file using tar or send it over the network using rsync, it will ...
2
votes
1answer
144 views

Looking for a “undo” for file with holes (GNU)

Linux supports sparse files aka 'file with holes' Note following commands alias mystat='stat -c "%n: %B*%b blocks %s bytes"' dd if=/dev/zero bs=1024k seek=4096 count=0 of=file-with-holes ls -l ...
6
votes
2answers
2k views

zerofree verbose returns what?

zerofree -v /dev/sda1 returned 123642/1860888/3327744. The man page does not explain what those numbers are: http://manpages.ubuntu.com/manpages/natty/man8/zerofree.8.html I found some code on ...
1
vote
1answer
1k views

Ext4 exhibits unexpected write latency variance vs. ext2

I have a latency sensitive application running on an embedded system, and I'm seeing some discrepancy between writing to a ext4 partition and an ext2 partition on the same physical device. ...
1
vote
1answer
516 views

What parts of the filesystem to tmpfs/squashfs/sparseimage for USB live system

I want to create a live system on a USB key / SDCard with Linux Mint 14 (KDE or Debian Edition) that stores data persistently. The USB key needs to have at least one fat32 partition. As storage for ...
2
votes
1answer
502 views

Stripping null byte blocks with a sparse file?

I've created a sparse file with dd. How do I copy contents of another file there, leaving all the zero blocks unallocated?
18
votes
3answers
3k views

How can you trim files using the command line?

What is the console equivalent of the following Python code: target = file("disk", "w") # create a file target.seek(2*1024*1024*1024) # skip to 2 GB target.write("\0") target.close() Maybe some ...
2
votes
1answer
964 views

Can QEMU/KVM+OCFS2 be configured to punch holes in sparse image files when files are deleted in the guest?

I'm using sparse raw image files for guest VMs and OCFS2 on the host (local-only, not clustered) for the reflink feature. I understand from Googling around that OCFS2 is one of the filesystems that ...
28
votes
4answers
4k views

Can a file that was originally sparse and then expanded be made sparse again?

I know that copying or transferring what was originally a sparse file without using a utility that understands sparse files will cause the 'holes' to be filled out. Is there a method or utility to ...
5
votes
3answers
1k views

from a sparse file to a block device over the network

If I have a sparse file representing a block device, then how can I efficiently restore the file over a network? Ideally, I'd use rsync -S host:file /dev/sdb, but rsync won't write to block devices. ...
16
votes
4answers
8k views

Creating an arbitrarily large “fake” file

I would like to create a special file similar to /dev/null or /dev/random, where the file doesn't actually exist but you can read from it all the same, except that I could actually set a cap on the ...
4
votes
1answer
3k views

Sparse files/file holes and unexpected block size

For my own learning, I've been playing around with creating files with file holes. I created a util that simply reads from stdin and writes to a file, but before writing to the file, it uses lseek to ...
3
votes
3answers
4k views

Is creating a sparse image using dd appropriate for backup/restore from a RAID?

I have two non-networked servers, each with 1.2TB of storage in a RAID5 config on an LSI MegaRaid 9240-8i controller. I have been trying without success to get Symantec Ghost to image one machine to ...
6
votes
1answer
2k views

Fragmentation and ext4

Suppose I have a sparse file F on a Linux ext4 filesystem, and process P1 is writing to a disjoint 50% subset of F while P2 writes to the other 50% of F. I would like to minimize fragmentation while ...
2
votes
1answer
4k views

What is the interaction of the rsync --size-only and --sparse options?

If I use rsync with the --size-only and --sparse options together, if a file already exists at the destination as a full file (otherwise the same) it's not overwritten. If I omit the --size-only ...
30
votes
5answers
21k views

Clone whole partition or hard drive to a sparse file

I like to clone a whole partition or a whole hard drive onto a larger external disk but like to create a sparse file. I often use dd for cloning, but it doesn't support sparse files. As a workaround I ...
10
votes
2answers
20k views

How to make ext4 filesystem sparse?

I am using Linux as guest OS in VirtualBox. I deleted huge number of files from its filesystem. Now i want to shrink the filesystem image file (vdi). The shrinking works by compressing filesystem ...
11
votes
3answers
3k views

Detailed sparse file information on Linux

I have a sparse file, in which only some blocks are allocated: ~% du -h --apparent-size example 100K example ~% du -h example 52K example I would like to know which blocks of the file are ...