At work we use sparse files as part of out Oracle VM environment for the guest disk images. After some questions from a colleague (which have since been answered) I am left with more questions about sparse files, and perhaps more widely about inode structure - reading the man pages of stat(2) and statfs(2) (on FreeBSD) I get the impression that I'd understand more readily if I knew more C, but alas my knowledge of C is minimal at best...
I understand that some of this is dependent on file system type. I'm mostly interested UFS on FreeBSD/Solaris and ext4 - ZFS would be a plus but I'm not going to hold out hope :)
I am using Solaris 10, FreeBSD 10.3, and CentOS 6.7 regularly. The commands here are being run on a CentOS 6.7 VM, but have been cross referenced with FreeBSD.
If possible, I'm interested in gaining an understanding from a POSIX viewpoint, and favouring FreeBSD over Linux if that isn't possible.
Consider the following set of commands:
printf "BIL" > /tmp/BIL
dd of=/tmp/sparse bs=1 count=0 seek=10
dd if=/tmp/BIL of=/tmp/sparse bs=1 count=3 seek=10
dd if=/tmp/BIL of=/tmp/sparse bs=1 count=3 seek=17
dd of=/tmp/sparse bs=1 count=0 seek=30
dd if=/tmp/BIL of=/tmp/sparse bs=1 count=3 seek=30
The file /tmp/BIL
should have the contents (in hex) of 4942 004c
, so when I hexdump
the file /tmp/sparse
I should see a smattering of this combination throughout:
%>hexdump sparse
0000000 0000 4942 004c 0000 0000 4942 004c 0000
0000010 4200 4c49 0000 0000 0000 0000 0000 4942
0000020 004c
0000021
%>cat sparse
BILBILBILBIL%
1. Why does the second occurrence of "BIL" appear out of order? i.e. 4200 4c49
rather than 4942 004c
? This was written by the third dd
command.
2. How does cat
and other tools know to print in the correct order?
Using ls
we can see the space allegedly used and the blocks allocated:
%>ls -ls /tmp/sparse
8.0K -rw-r--r--. 1 bil bil 33 May 26 14:17 /tmp/sparse
We can see that the alleged size is 33 bytes, but allocated size is 8 kilobytes (file system block size is 4K).
3. How do programs like ls
discern between the "alleged" size and the allocated size?
I wondered if the "alleged" figure stored in the inode while the allocated size was calculated by walking the direct and indirect blocks - though this cannot be correct since calculation via walking would take time and tools such as ls
return quickly, even for very large files.
4. What tools can I use to interrogate inode information?
I know of stat
, but it doesn't seem to print out the values of all of the fields in an inode...
5. Is there a tool where I can walk the direct and indirect blocks?
It would be interesting to see each address on disk, and the contents to gain a bit more understanding of how data is stored
If I run the following command after the others above, the file /tmp/sparse
is truncated:
%>dd of=/tmp/sparse bs=1 count=0 seek=5
%>hexdump sparse
0000000 0000 4942 004c
0000005
6. Why does dd
truncate my file and can dd
or another tool write into the middle of a file?
Lastly, sparse files seem like a Good Idea for preallocating space, but there doesn't appear to be file system or operating system level assurances that the a command won't truncate or arbitrarily grow the file.
7. Are there mechanisms to prevent sparse files be shrunk/grown? And if not, why are sparse files useful?
While each question above could possibly be a separate SO question, I cannot dissect them as they are all related to the underlying understanding.