8

I am hoping I understand sparse file concept. I also do know cp command's --sparse=...

However when googling for the practical applications I found ambiguous statements about how transparent it is for an application which reads/writes files using the common operating system file I/O API (I mean not in the extreme low level, just fopen(), fclose() etc)

It is not clean when reading blogs, explanations what talk about how an application for example a test editor "ruins" a sparse file by explicitly writing zeros to it. I thought this is the point, that if there is a sparse file, and the application writes zeros, that will not be physically stored. The application does not have to know about this, and does not have to deal with gaps and such a things, that is the responsibility of the file system.

Question

Suppose there is an existing file which is sparse. Will it completely transparent for the application or not? Say there is an 1G sparse file, which have very first byte is non zero, all other bytes are zero. When a "common" application opens that file, I suppose it can open it, an will see as its length is 1G, and can seek to the middle (0.5G), as it were not sparse, can write a non zero byte to the middle, the save, close and it will remain sparse on the file system, will not it?

Will a file 'automatically' sparse? I mean, an application simply creates a file, then writes a bunch of zeros, then writes, will is sparse or not? If not, what an application should do to create that file as sparse?

4
  • IMO people should avoid creating/using sparse files unless they absolutely need them. Such files result in insane amount of FS fragmentation and extra work from the FS driver. There are many more disadvantages and pitfalls: wiki.archlinux.org/title/sparse_file Commented Jan 26, 2023 at 8:05
  • @ArtemS.Tashkinov I was under the impression that modern filesystems like ext4 handled sparse files without causing heavy fragmentation. If I recall, using fallocate to create a large file, writing to random locations, then using filefrag to determine the number of fragments shows a relatively low value. If anything, not using sparse files and merely appending to a file when you need rather than preallocating it is going to cause more fragmentation.
    – forest
    Commented Jan 26, 2023 at 11:26
  • @forest that all works well if you have >80% free space (looks crazy but that's what it is). If you're under 60% and have lots of files, it all goes downhill fast. ext4 cannot defragment free space, only individual files and fragmentation quickly becomes an insurmountable issue. AFAIK xfs is the only native Linux FS which can defragment everything (files and free space). Commented Jan 26, 2023 at 11:29
  • @ArtemS.Tashkinov BTRFS can as well, though it’s a bit more complicated than with XFS (it requires multiple distinct commands run in a particular order, and it’s non-obvious how to do it if you don’t understand how the filesystem works under the hood). That said, the performance difference usually does not matter unless you’re doing something that’s super-dependent on I/O performance or dealing with particularly slow storage. It’s worth worrying about it on something like a database server or file server, but not on a typical client system. Commented Jan 26, 2023 at 20:50

1 Answer 1

12

In the majority of cases, sparse files are transparent to programs and programs shouldn’t need to care whether the files they are handling are sparse or not.

Sparse files are created by skipping blocks, creating holes in the file, not by writing zeroes. If a program seeks past the end of a file, then causes the file’s size to be updated (by writing more data), or “truncates” a file to a length longer than the data it contains, the resulting file will be sparse (if the underlying file system supports sparse files).

Sparse files are transparent on reads (holes are read as zeroes), but not on writes: writing any data to a block forces it to be allocated and ultimately written. In particular, this means that writing to a file without changing its length can fail if the underlying file system is full. This doesn’t mean that code writing files should have a special case for sparse files; it just means that errors should be handled on all writes (as you’d expect).

The Linux implementation of lseek provides extensions to allow holes in files to be analysed.

Files aren’t automatically made sparse; that’s why GNU cp’s --sparse option exists — it configures cp to detect runs of zeroes itself and produce holes in the target instead of writing them. If files were automatically made sparse, this wouldn’t be necessary.

In your scenario, a 1G file with one byte of data at the start will have one block on disk, containing that byte followed by however many zeroes fit in a block. The rest of the file will be one large hole. Writing a zero to the middle of the file will allocate one block and fill it with zeroes. The file will then contain one block, a near-0.5G hole, another block, and another hole.

12
  • many thx. So if I understand correctly, in case the application writes into a sparse file it must aware it is a sparse file, so must handle it accordingly. (sparse files are not transparent file system feature for writing, just reading.) Commented Jan 26, 2023 at 7:46
  • @g.pickardou No, it's completely transparent to the application. Whether the opened file is on a filesystem that doesn't support sparse files or one that does, the semantics will be identical.
    – forest
    Commented Jan 26, 2023 at 7:47
  • 1
    @g.pickardou as forest says, programs don’t need to be aware that a file they’re writing to is sparse; the main difference is that developers who aren’t aware that sparse files exist might not write the error-handling code correctly. A program which writes to files needs to handle “out of space” errors whether it’s appending to the file or not. Outside of very specific cases (e.g. cp --sparse), programs should never try to determine whether a file is sparse or not. Commented Jan 26, 2023 at 7:50
  • 2
    find . -file f -printf '%S %p\n' will list files with an estimate of their sparseness (estimated from the blocks actually allocated and the total length). Commented Jan 26, 2023 at 9:42
  • 1
    @g.pickardou it should never make the application crash, unless the application mis-handles running out of disk space and that becomes an issue (but that’s not tied to sparse files). It is often counter-productive to sparsify a file with an application that doesn’t handle sparse files itself. For MySQL, you should use storage engine settings instead; InnoDB for example will sparsify its own files if compression is enabled (see dev.mysql.com/doc/refman/8.0/en/innodb-page-compression.html and mariadb.com/kb/en/innodb-page-compression/…). Commented Jan 27, 2023 at 8:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .