Questions tagged [sparse-files]
The sparse-files tag has no usage guidance.
70
questions
-1
votes
1
answer
88
views
Truncate command not creating a hole
I am trying to create a file with hole using the truncate command. I read up in some posts and one of the answers in this post says to use truncate command. Filesystem used is btrfs. This is the ...
8
votes
1
answer
785
views
How transparent are sparse files for applications?
I am hoping I understand sparse file concept.
I also do know cp command's --sparse=...
However when googling for the practical applications I found ambiguous statements about how transparent it is for ...
- 236
0
votes
1
answer
308
views
How to mount a sparsed disk image
I have an image of the mSATA SSD disk of a PC. The disk contains the operating system and has a capacity of 512GB.
I do not have that free storage, so I have cloned the disk into an image with dd ...
- 1,158
1
vote
1
answer
183
views
Extracting text with expressions from a text file using grep
I have the following 2 lines in my text file and wanted to calculate duration between those lines X & Y in minutes.
line X: 18.05.2022 13:54:52 [ INFO]: Starting Component 'OWN_FUNDS_RULES' (5/15)...
- 127
18
votes
2
answers
2k
views
How do I uncompress a file with lots of zeroes as a sparse file?
I have a compressed raw image of a very large hard drive created using cat /dev/sdx | xz > image.xz. However, the free space in the drive was zeroed before this operation, and the image consists ...
0
votes
1
answer
136
views
Shrunk virtual disk file, now reported file size doesn't match
I originally had a single virtual disk file (.qcow2) of roughly 3.5 TiB and I decided to "group" all similar data on its own virtual disk file, for various reasons. Obviously that meant I ...
- 111
1
vote
0
answers
1k
views
Moving sparse files
cp not only auto-detects sparse files but also has --sparse=always option.
I wonder if mv works with the same mechanism or the safe way for moving sparse files is
cp && rm
Or is there any ...
- 1,919
0
votes
2
answers
4k
views
Tell if a file is a "sparse file"
There is an existing question for "Finding all sparse files", and I think I could use it, but the scripts were too complex to understand. I don't think the GUI of Ubuntu file manager ...
- 1,227
2
votes
1
answer
245
views
Sparse, bootable Linux images? [closed]
I've set up a new Linux workstation and would like to create a backup image in order to save time in case something breaks. This is the situation:
Workstation: 512 GB SSD, 12 GB used, ext4 file ...
- 319
0
votes
3
answers
747
views
Filter the rows of a file based on the number of columns that have =0.00000000
I would like to filter the rows of a file (e.g. file.txt below) based on the number of columns, after column 5, that have =0.00000000.
The I/O below shows an example for filtering rows that have ...
- 559
4
votes
0
answers
339
views
Is it safe to enable Direct I/O for a loop device backed by a sparse file?
I'd like to have a sparse file, and mount it as a filesystem. It'd be great if the performance overhead is negligible, but only if that doesn't come with a data-loss risk! Is it safe to mount the ...
- 141
2
votes
1
answer
858
views
Disable sparse file support on ext4
Is it possible to completely disable sparse file support in an ext4 file system?
The purpose is to avoid disk fragmentation.
Bonus "no-points" if the solution allows the file system to allocate the ...
- 125
2
votes
1
answer
675
views
Is it possible to delete sections of a sparse file?
I'm working with memory mapped files. Each file is allocated using truncate (see man 2 truncate). These are then memory mapped and the application writes entries into various sections of the files ...
- 18.4k
11
votes
2
answers
2k
views
Make grown extracted tar file small again
I packed and compressed a folder to a .tar.gz archive.
After unpacking it was nearly twice as big.
du -sh /path/to/old/folder = 263M
du -sh /path/to/extracted/folder = 420M
I searched a lot ...
- 113
21
votes
4
answers
36k
views
Is there a reason why /var/log/lastlog is a huge sparse file (1.1TB)?
I have read some question, that ask advice how to rsync sparse files efficiently mentioning the files /var/log/lastlog and /var/log/faillog. Indeed I myself have stumpled over those files being an "...
- 14.1k
4
votes
1
answer
2k
views
How to view contents of a sparse file?
The size of "file" is showing 15TB when i check the size with ls -l
ls -l
total 16
-rw-r--r-- 1 root root 15393162788865 May 30 13:41 file
when i check the size of the "file" with du command, it ...
- 43
1
vote
0
answers
63
views
Suitable filesystem for heavily fragmented files
I often do data recovery jobs using ddrescue or similar programs. The disk image is written sparsely, because often only parts of the disk is being imaged (using ddrescue domain files and the like). ...
- 683
1
vote
1
answer
1k
views
Is there some program that can copy sparse file (/var/log/lastlog) over ssh as fast as cp (on local pc)?
I'm backing ip my server via rsync over ssh but /var/log/lastlog file is 1.2G (it takes only 24K on the hdd).
On a local machine cp can copy it for no time (a few ms) but rsync requires reading the ...
- 35
2
votes
1
answer
558
views
How to change zram sector size?
I have cloned a disk into a sparse file that is about 80G but indeed requires only about 12G, even not compressed it fits on my memory, but in order to save some resources I want to use zram:
sudo ...
- 606
0
votes
0
answers
276
views
Shrink LVM2 Thin Volume
I would like to shrink the space required by a thin volume used as a disk for a virtual machine (Windows). Unfortunately, the underlying disk is a USB-attached SSD and therefore mounting the NTFS ...
- 41
1
vote
0
answers
735
views
tar takes a long time before passing data to gzip
What I want to know, is what is tar doing at the start, before it starts passing data on to gzip? Can I make it skip that step?
I'm writing a script to run on my Synology NAS box (running DSM 6.2.1-...
- 704
1
vote
0
answers
349
views
loop device not matching file size
I have a sparse backing file (NTFS) (made via dd seek=) that's a partial image of a damaged disk. The size as given by ls matches the original disk. However, when looping it, /proc/partitions shows it ...
- 683
5
votes
2
answers
2k
views
Can losetup be made efficient with sparse files?
So my setup is like this.
$ truncate -s 1T volume
$ losetup -f --show volume
/dev/loop0
$ mkfs.ext4 /dev/loop0
$ ls -sh volume
1.1G volume
$ mount /dev/loop0 /mnt/loop
Now I have a 1.1TB volume, as ...
- 395
1
vote
1
answer
805
views
Is it possible to zero the free space on a LUKS volume?
I want to have a LUKS volume as a loopback device pointing at a sparse file. Lets say I put a 20gb file on it, then later I delete it. How can I zero that area on the device itself so I can fallocate -...
- 395
3
votes
1
answer
5k
views
rsync: is possible to copy only the used space of qcow2 or raw image?
Simple question.
I have a 200G qcow2 or raw disk image, but of 200G only 30G are used.
If I do
rsync -avP
copy all the 200G on usb disk or network and it wast time,space and network bandwidth.
Is ...
- 12.4k
8
votes
1
answer
11k
views
qcow2 actual size
I am little bit confused with the real size of qcow2 files.
ls -alh VMs/ubuntu-mini.qcow2
-rw------- 1 root root 21G mar 31 23:15 VMs/ubuntu-mini.qcow2
du -h VMs/ubuntu-mini.qcow2
2,7G VMs/ubuntu-...
- 113
10
votes
2
answers
4k
views
What is the difference between `fallocate --dig-holes` and `fallocate --punch-hole` in Linux?
I totally understand that --dig-holes creates a sparse file in-place. That is, if the file has holes --dig-holes options removes those holes:
Let's take it in a very simplified way, let's say we ...
- 974
13
votes
1
answer
9k
views
Are there any disadvantages of `cp --sparse=always`?
Is there any reason not to use use --sparse=always with every invocation to cp?
info cp says:
‘--sparse=WHEN’
A “sparse file” contains “holes”—a sequence of zero bytes that does
not occupy ...
- 29.4k
0
votes
1
answer
702
views
Can we allocate file holes to another file?
With reference to the Advanced Programming in the Unix Environment the file holes are created when we try to seek past the end of the file and write something there.For example,
int x = lseek(fd,...
- 133
2
votes
1
answer
2k
views
What are functions to manipulate sparse files under Linux?
What are functions to manipulate sparse files under Linux? (let's say in C, notes about other systems highly welcome)
e.g.:
make hole inside of file by removing part of its inside
investigate ...
2
votes
1
answer
752
views
Files created by fallocate compared to normal text files
I am testing S3 command line tool performances at the moment and I am testing them using files I have created with fallocate. Im measuring the time different tools take to upload, say, 200 files of ...
15
votes
1
answer
2k
views
What could explain this strange sparse file handling of/in tmpfs?
On my ext4 filesystem partition I can run the following code:
fs="/mnt/ext4"
#create sparse 100M file on ${fs}
dd if=/dev/zero \
of=${fs}/sparse100M conv=sparse seek=$((100*2*1024-1)) count=1 2&...
- 14.1k
7
votes
2
answers
4k
views
rsync of sparse qemu image increases disk size
I have a sparse raw qemu image that I want to transfer to another server. qemu-img info gives me:
image: sparse.img
file format: raw
virtual size: 50G
disk size: 16G
I transfer it with:
rsync -azhP ...
- 518
2
votes
1
answer
397
views
Output of ls command confusing for KVM raw image on XFS
After creating a new VM on a raw image using virt-install, I tried to verify that the image was a sparse file, but was unable to determine using the ls command.
ls -ls myvm.img
1048576000 -rw------- ...
- 518
15
votes
3
answers
1k
views
How do I output "sparseness" of a file?
How do I output how much of file nominal size is actually filled with data? Like vmtouch shows how much of file is currently in memory...
I expect the workflow to be like this:
$ fallocate -l ...
- 5,598
0
votes
2
answers
519
views
parsing using loop and assigning row numbers to file
It is a bit complicated. i have a configuration file. it looks something like this.
customer="airtel"
model_name=fast
programmer_typ_of="dev"
Now the 1st column contains all the keys and whatever is ...
8
votes
2
answers
931
views
Why is fragmentation level so huge in files that contain other filesystems?
I've just found out what sparse files are and wanted to conduct some experiments on them. On wiki you can read that the files can get fragmented easily. I wanted to check how bad that is. I created a ...
- 10.4k
7
votes
2
answers
5k
views
Understanding sparse files, dd, seek, inode block structure
At work we use sparse files as part of out Oracle VM environment for the guest disk images. After some questions from a colleague (which have since been answered) I am left with more questions about ...
- 3,426
1
vote
1
answer
558
views
Create and use sparse file as non-privileged user
Here is my setup:
I am non-privileged user (non-root).
I cannot use sudo, nor I can use any other command to switch to root (e.g. su), or run commands as root.
As a consequence, I cannot modify the /...
- 501
3
votes
2
answers
6k
views
How to get number of files in a 7z
I want to get the number of files (included directories) contained in the root of a 7z archive, but the 7z command on Linux don’t bring a simple way to only get the number and noting else.
For ...
- 1,150
0
votes
1
answer
420
views
Can a sparse file go beyond the ram size?
Since the physical size of the file would be less than the logical file size, is it possible to create a sparse file with a size bigger than the available ram?
- 307
1
vote
1
answer
2k
views
rsync generating a destination directory of bigger size than source?
We are copying our data from source server to destination server using rsync :
rsync -av --progress --sparse user@destination server
After rsync, we observed that destination directory is bigger in ...
- 21
1
vote
0
answers
116
views
xfs: writing to 17TB offset doesn't actually write it to specified offset
I have a situation wherein I want to write a few GBs at a very large offset - say 17TB. But when I use the dd command to do so, I notice that the filesystem is doing some kind of optimization - sparse ...
- 11
1
vote
0
answers
2k
views
Creating a sparse (automatically expanding and contracting) swap file with hole-punching support
I have this crazy idea, and want to know the pitfalls of such an approach (or whether it can even work with no deadlocks.. etc).
The goal is to create an automatically expanding and contracting ...
- 509
6
votes
2
answers
3k
views
How do I punch a hole (FALLOC_FL_PUNCH_HOLE) in a file from command line?
There is fallocate(2) system call to make sparse files by erasing a "holes" in them. Is there a command line tool for it?
Expecting something like:
makehole myfile.dat --offset=10G --length=1G
- 5,598
1
vote
1
answer
146
views
file (vdi) increase (at double) size after split and then cat operation
I'd splitted a vdi file with 7 gigabytes and then joined with cat command. The file works normally on VBox, but get 14 gigabytes after be re-joined. The exact operation was:
$ split -- bytes=2000M ...
- 165
3
votes
1
answer
482
views
"Ghost" files in ext4? (ktorrent allocated disk space)
I'm downloading with KTorrent, and have noticed that when I select "allocate disk space before starting torrent", it creates all the files at start, presumably empty, to avoid fragmentation.
When I'm ...
- 1,222
2
votes
1
answer
865
views
What is the correct syntax for cloning a ntfs partition to a sparse file on another ntfs drive?
I have tried ddrescue -v -v -S /dev/sda2 /media/mint/otherhdd/img /media/mint/otherhdd/log and ddrescue -v -v -d -S /dev/sda2 /media/mint/otherhdd/img /media/mint/otherhdd/log but it does not work ...
- 21
3
votes
3
answers
2k
views
From a sparse file to a block device
I have a system image sparse file which actual size is only a few MBs but the "apparent size" is about 1GB. I'm trying to write it to a block device efficiently (without the holes). Here are some ...
13
votes
1
answer
10k
views
Converting sparse file to non-sparse in place
On Linux, given a sparse file, how to make it non-sparse, in place?
It could be copied with cp --sparse=never ..., but if the file is say 10G and the hole is 2G
(that is the allocated space is 8G), ...
- 233