Questions tagged [sparse-files]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1 vote
1 answer
73 views

Extracting text with expressions from a text file using grep

I have the following 2 lines in my text file and wanted to calculate duration between those lines X & Y in minutes. line X: 18.05.2022 13:54:52 [ INFO]: Starting Component 'OWN_FUNDS_RULES' (5/15)...
user avatar
18 votes
2 answers
2k views

How do I uncompress a file with lots of zeroes as a sparse file?

I have a compressed raw image of a very large hard drive created using cat /dev/sdx | xz > image.xz. However, the free space in the drive was zeroed before this operation, and the image consists ...
user avatar
0 votes
1 answer
51 views

Shrunk virtual disk file, now reported file size doesn't match

I originally had a single virtual disk file (.qcow2) of roughly 3.5 TiB and I decided to "group" all similar data on its own virtual disk file, for various reasons. Obviously that meant I ...
user avatar
  • 11
1 vote
0 answers
582 views

Moving sparse files

cp not only auto-detects sparse files but also has --sparse=always option. I wonder if mv works with the same mechanism or the safe way for moving sparse files is cp && rm Or is there any ...
user avatar
  • 1,743
0 votes
2 answers
2k views

Tell if a file is a "sparse file"

There is an existing question for "Finding all sparse files", and I think I could use it, but the scripts were too complex to understand. I don't think the GUI of Ubuntu file manager ...
user avatar
1 vote
1 answer
158 views

Sparse, bootable Linux images? [closed]

I've set up a new Linux workstation and would like to create a backup image in order to save time in case something breaks. This is the situation: Workstation: 512 GB SSD, 12 GB used, ext4 file ...
user avatar
  • 1
0 votes
3 answers
361 views

Filter the rows of a file based on the number of columns that have =0.00000000

I would like to filter the rows of a file (e.g. file.txt below) based on the number of columns, after column 5, that have =0.00000000. The I/O below shows an example for filtering rows that have ...
user avatar
  • 559
2 votes
0 answers
176 views

Is it safe to enable Direct I/O for a loop device backed by a sparse file?

I'd like to have a sparse file, and mount it as a filesystem. It'd be great if the performance overhead is negligible, but only if that doesn't come with a data-loss risk! Is it safe to mount the ...
user avatar
  • 121
0 votes
1 answer
452 views

Disable sparse file support on ext4

Is it possible to completely disable sparse file support in an ext4 file system? The purpose is to avoid disk fragmentation. Bonus "no-points" if the solution allows the file system to allocate the ...
user avatar
2 votes
1 answer
384 views

Is it possible to delete sections of a sparse file?

I'm working with memory mapped files. Each file is allocated using truncate (see man 2 truncate). These are then memory mapped and the application writes entries into various sections of the files ...
user avatar
10 votes
2 answers
2k views

Make grown extracted tar file small again

I packed and compressed a folder to a .tar.gz archive. After unpacking it was nearly twice as big. du -sh /path/to/old/folder = 263M du -sh /path/to/extracted/folder = 420M I searched a lot ...
user avatar
11 votes
4 answers
23k views

Is there a reason why /var/log/lastlog is a huge sparse file (1.1TB)?

I have read some question, that ask advice how to rsync sparse files efficiently mentioning the files /var/log/lastlog and /var/log/faillog. Indeed I myself have stumpled over those files being an "...
user avatar
5 votes
1 answer
1k views

How to view contents of a sparse file?

The size of "file" is showing 15TB when i check the size with ls -l ls -l total 16 -rw-r--r-- 1 root root 15393162788865 May 30 13:41 file when i check the size of the "file" with du command, it ...
user avatar
  • 53
1 vote
0 answers
46 views

Suitable filesystem for heavily fragmented files

I often do data recovery jobs using ddrescue or similar programs. The disk image is written sparsely, because often only parts of the disk is being imaged (using ddrescue domain files and the like). ...
user avatar
1 vote
1 answer
990 views

Is there some program that can copy sparse file (/var/log/lastlog) over ssh as fast as cp (on local pc)?

I'm backing ip my server via rsync over ssh but /var/log/lastlog file is 1.2G (it takes only 24K on the hdd). On a local machine cp can copy it for no time (a few ms) but rsync requires reading the ...
user avatar
2 votes
1 answer
422 views

How to change zram sector size?

I have cloned a disk into a sparse file that is about 80G but indeed requires only about 12G, even not compressed it fits on my memory, but in order to save some resources I want to use zram: sudo ...
user avatar
0 votes
0 answers
249 views

Shrink LVM2 Thin Volume

I would like to shrink the space required by a thin volume used as a disk for a virtual machine (Windows). Unfortunately, the underlying disk is a USB-attached SSD and therefore mounting the NTFS ...
user avatar
1 vote
0 answers
432 views

tar takes a long time before passing data to gzip

What I want to know, is what is tar doing at the start, before it starts passing data on to gzip? Can I make it skip that step? I'm writing a script to run on my Synology NAS box (running DSM 6.2.1-...
user avatar
1 vote
0 answers
247 views

loop device not matching file size

I have a sparse backing file (NTFS) (made via dd seek=) that's a partial image of a damaged disk. The size as given by ls matches the original disk. However, when looping it, /proc/partitions shows it ...
user avatar
2 votes
2 answers
1k views

Can losetup be made efficient with sparse files?

So my setup is like this. $ truncate -s 1T volume $ losetup -f --show volume /dev/loop0 $ mkfs.ext4 /dev/loop0 $ ls -sh volume 1.1G volume $ mount /dev/loop0 /mnt/loop Now I have a 1.1TB volume, as ...
user avatar
  • 303
1 vote
1 answer
554 views

Is it possible to zero the free space on a LUKS volume?

I want to have a LUKS volume as a loopback device pointing at a sparse file. Lets say I put a 20gb file on it, then later I delete it. How can I zero that area on the device itself so I can fallocate -...
user avatar
  • 303
1 vote
1 answer
4k views

rsync: is possible to copy only the used space of qcow2 or raw image?

Simple question. I have a 200G qcow2 or raw disk image, but of 200G only 30G are used. If I do rsync -avP copy all the 200G on usb disk or network and it wast time,space and network bandwidth. Is ...
user avatar
  • 11.1k
6 votes
1 answer
9k views

qcow2 actual size

I am little bit confused with the real size of qcow2 files. ls -alh VMs/ubuntu-mini.qcow2 -rw------- 1 root root 21G mar 31 23:15 VMs/ubuntu-mini.qcow2 du -h VMs/ubuntu-mini.qcow2 2,7G VMs/ubuntu-...
user avatar
  • 83
9 votes
2 answers
3k views

What is the difference between `fallocate --dig-holes` and `fallocate --punch-hole` in Linux?

I totally understand that --dig-holes creates a sparse file in-place. That is, if the file has holes --dig-holes options removes those holes: Let's take it in a very simplified way, let's say we ...
user avatar
  • 864
13 votes
1 answer
7k views

Are there any disadvantages of `cp --sparse=always`?

Is there any reason not to use use --sparse=always with every invocation to cp? info cp says: ‘--sparse=WHEN’ A “sparse file” contains “holes”—a sequence of zero bytes that does not occupy ...
user avatar
  • 25.4k
0 votes
1 answer
551 views

Can we allocate file holes to another file?

With reference to the Advanced Programming in the Unix Environment the file holes are created when we try to seek past the end of the file and write something there.For example, int x = lseek(fd,...
user avatar
2 votes
1 answer
2k views

What are functions to manipulate sparse files under Linux?

What are functions to manipulate sparse files under Linux? (let's say in C, notes about other systems highly welcome) e.g.: make hole inside of file by removing part of its inside investigate ...
user avatar
2 votes
1 answer
599 views

Files created by fallocate compared to normal text files

I am testing S3 command line tool performances at the moment and I am testing them using files I have created with fallocate. Im measuring the time different tools take to upload, say, 200 files of ...
user avatar
15 votes
1 answer
1k views

What could explain this strange sparse file handling of/in tmpfs?

On my ext4 filesystem partition I can run the following code: fs="/mnt/ext4" #create sparse 100M file on ${fs} dd if=/dev/zero \ of=${fs}/sparse100M conv=sparse seek=$((100*2*1024-1)) count=1 2&...
user avatar
6 votes
2 answers
4k views

rsync of sparse qemu image increases disk size

I have a sparse raw qemu image that I want to transfer to another server. qemu-img info gives me: image: sparse.img file format: raw virtual size: 50G disk size: 16G I transfer it with: rsync -azhP ...
user avatar
  • 470
2 votes
1 answer
358 views

Output of ls command confusing for KVM raw image on XFS

After creating a new VM on a raw image using virt-install, I tried to verify that the image was a sparse file, but was unable to determine using the ls command. ls -ls myvm.img 1048576000 -rw------- ...
user avatar
  • 470
15 votes
3 answers
1k views

How do I output "sparseness" of a file?

How do I output how much of file nominal size is actually filled with data? Like vmtouch shows how much of file is currently in memory... I expect the workflow to be like this: $ fallocate -l ...
user avatar
  • 5,186
0 votes
2 answers
342 views

parsing using loop and assigning row numbers to file

It is a bit complicated. i have a configuration file. it looks something like this. customer="airtel" model_name=fast programmer_typ_of="dev" Now the 1st column contains all the keys and whatever is ...
user avatar
8 votes
2 answers
829 views

Why is fragmentation level so huge in files that contain other filesystems?

I've just found out what sparse files are and wanted to conduct some experiments on them. On wiki you can read that the files can get fragmented easily. I wanted to check how bad that is. I created a ...
user avatar
7 votes
2 answers
4k views

Understanding sparse files, dd, seek, inode block structure

At work we use sparse files as part of out Oracle VM environment for the guest disk images. After some questions from a colleague (which have since been answered) I am left with more questions about ...
user avatar
  • 3,266
1 vote
1 answer
529 views

Create and use sparse file as non-privileged user

Here is my setup: I am non-privileged user (non-root). I cannot use sudo, nor I can use any other command to switch to root (e.g. su), or run commands as root. As a consequence, I cannot modify the /...
user avatar
2 votes
2 answers
5k views

How to get number of files in a 7z

I want to get the number of files (included directories) contained in the root of a 7z archive, but the 7z command on Linux don’t bring a simple way to only get the number and noting else. For ...
user avatar
  • 960
0 votes
1 answer
315 views

Can a sparse file go beyond the ram size?

Since the physical size of the file would be less than the logical file size, is it possible to create a sparse file with a size bigger than the available ram?
user avatar
1 vote
1 answer
2k views

rsync generating a destination directory of bigger size than source?

We are copying our data from source server to destination server using rsync : rsync -av --progress --sparse user@destination server After rsync, we observed that destination directory is bigger in ...
user avatar
1 vote
0 answers
108 views

xfs: writing to 17TB offset doesn't actually write it to specified offset

I have a situation wherein I want to write a few GBs at a very large offset - say 17TB. But when I use the dd command to do so, I notice that the filesystem is doing some kind of optimization - sparse ...
user avatar
1 vote
0 answers
2k views

Creating a sparse (automatically expanding and contracting) swap file with hole-punching support

I have this crazy idea, and want to know the pitfalls of such an approach (or whether it can even work with no deadlocks.. etc). The goal is to create an automatically expanding and contracting ...
user avatar
5 votes
2 answers
3k views

How do I punch a hole (FALLOC_FL_PUNCH_HOLE) in a file from command line?

There is fallocate(2) system call to make sparse files by erasing a "holes" in them. Is there a command line tool for it? Expecting something like: makehole myfile.dat --offset=10G --length=1G
user avatar
  • 5,186
1 vote
1 answer
132 views

file (vdi) increase (at double) size after split and then cat operation

I'd splitted a vdi file with 7 gigabytes and then joined with cat command. The file works normally on VBox, but get 14 gigabytes after be re-joined. The exact operation was: $ split -- bytes=2000M ...
user avatar
3 votes
1 answer
416 views

"Ghost" files in ext4? (ktorrent allocated disk space)

I'm downloading with KTorrent, and have noticed that when I select "allocate disk space before starting torrent", it creates all the files at start, presumably empty, to avoid fragmentation. When I'm ...
user avatar
  • 1,182
2 votes
1 answer
733 views

What is the correct syntax for cloning a ntfs partition to a sparse file on another ntfs drive?

I have tried ddrescue -v -v -S /dev/sda2 /media/mint/otherhdd/img /media/mint/otherhdd/log and ddrescue -v -v -d -S /dev/sda2 /media/mint/otherhdd/img /media/mint/otherhdd/log but it does not work ...
user avatar
  • 21
3 votes
3 answers
1k views

From a sparse file to a block device

I have a system image sparse file which actual size is only a few MBs but the "apparent size" is about 1GB. I'm trying to write it to a block device efficiently (without the holes). Here are some ...
user avatar
13 votes
1 answer
8k views

Converting sparse file to non-sparse in place

On Linux, given a sparse file, how to make it non-sparse, in place? It could be copied with cp --sparse=never ..., but if the file is say 10G and the hole is 2G (that is the allocated space is 8G), ...
user avatar
  • 233
3 votes
1 answer
3k views

Linux: How to calculate/find the slackspace of a file?

How would I go about finding the slackspace of a file, for instance. fls -o 63 /images/diskimg.dd istat -o 63 /images/diskimg.dd 31 icat -o 63 /images/diskimg.dd | xxd | less In which of these ...
user avatar
  • 31
8 votes
1 answer
369 views

How to display the non-sparse parts of a sparse file?

Imagine a file created with: truncate -s1T file echo test >> file truncate -s2T file I now have a 2 tebibyte file (that occupies 4kiB on disk), with "test\n" written in the middle. How would ...
user avatar
9 votes
2 answers
8k views

How can I speed up operations on sparse files with tar, gzip, rsync?

I have a sparse file. (du -h reports 3G and du -h --apparent-size reports 100G.) So far, so good. Now, when I want to compress the file using tar or send it over the network using rsync, it will ...
user avatar
  • 1,609