How transparent are sparse files for applications?

Question

I am hoping I understand sparse file concept. I also do know cp command's --sparse=...

However when googling for the practical applications I found ambiguous statements about how transparent it is for an application which reads/writes files using the common operating system file I/O API (I mean not in the extreme low level, just fopen(), fclose() etc)

It is not clean when reading blogs, explanations what talk about how an application for example a test editor "ruins" a sparse file by explicitly writing zeros to it. I thought this is the point, that if there is a sparse file, and the application writes zeros, that will not be physically stored. The application does not have to know about this, and does not have to deal with gaps and such a things, that is the responsibility of the file system.

Question

Suppose there is an existing file which is sparse. Will it completely transparent for the application or not? Say there is an 1G sparse file, which have very first byte is non zero, all other bytes are zero. When a "common" application opens that file, I suppose it can open it, an will see as its length is 1G, and can seek to the middle (0.5G), as it were not sparse, can write a non zero byte to the middle, the save, close and it will remain sparse on the file system, will not it?

Will a file 'automatically' sparse? I mean, an application simply creates a file, then writes a bunch of zeros, then writes, will is sparse or not? If not, what an application should do to create that file as sparse?

Answer 1

In the majority of cases, sparse files are transparent to programs and programs shouldn’t need to care whether the files they are handling are sparse or not.

Sparse files are created by skipping blocks, creating holes in the file, not by writing zeroes. If a program seeks past the end of a file, then causes the file’s size to be updated (by writing more data), or “truncates” a file to a length longer than the data it contains, the resulting file will be sparse (if the underlying file system supports sparse files).

Sparse files are transparent on reads (holes are read as zeroes), but not on writes: writing any data to a block forces it to be allocated and ultimately written. In particular, this means that writing to a file without changing its length can fail if the underlying file system is full. This doesn’t mean that code writing files should have a special case for sparse files; it just means that errors should be handled on all writes (as you’d expect).

The Linux implementation of lseek provides extensions to allow holes in files to be analysed.

Files aren’t automatically made sparse; that’s why GNU cp’s --sparse option exists — it configures cp to detect runs of zeroes itself and produce holes in the target instead of writing them. If files were automatically made sparse, this wouldn’t be necessary.

In your scenario, a 1G file with one byte of data at the start will have one block on disk, containing that byte followed by however many zeroes fit in a block. The rest of the file will be one large hole. Writing a zero to the middle of the file will allocate one block and fill it with zeroes. The file will then contain one block, a near-0.5G hole, another block, and another hole.