22

As I understood, "sparse file" means that the file may have 'gaps' so the actual used data may be smaller than the logical file size.

How do Linux file systems save files on disk? I'm mainly interested in ext4. But:

  1. Can a file be saved not sequentially on disk? By that, I mean that part of the file is located at physical address X and the next part at physical address Y which isn't close to X + offset).
  2. Can I somehow control the file sequentiality?
    I want to allocate a file of 10GB. I want it to be sequential on disk and not divided between different offsets.
  3. Does it act differently between the different types?
  • You might want to read ext4.wiki.kernel.org/index.php/Main_Page – roaima Feb 13 '17 at 12:46
  • 1
    Perhaps, if I understand your intention correctly, you would be more interested in lower-level API, where you work with storage devices w/o having to go through the file-system layer. Your entry-point then could be the dmsetup program, an interface to device mapper. This may be a good choice if you are planning a database-like storage. – wvxvw Feb 13 '17 at 16:48
  • 4
    This is an implementation detail of the filesystem. Almost all filesystems do fragment files by default; only iso9660 and romfs are incapable of doing that and require continuous storage (of these I can list off-head). – mirabilos Feb 13 '17 at 17:44
  • 2
    whether the file is contiguous on disk or not, data read/write will always be contiguous unless you do a seek to another part of the file. So why do you care about this? Unless fragmentation is a serious problem that affects performance – phuclv Feb 14 '17 at 8:41
  • 3
    @hudac one thing to keep in mind is that contiguous is not all that useful in practice. The easy one is flash where fragmentation is not a big deal, but on a spinning platter you still might not benefit from contiguous data. On a spinning platter you need to think about your access patterns and where the data is. If you need the sector that just passed under the head you have to wait for it to come fully around again. To get the best results you want to stagger the data so that it is "close" when it needs to be read. Increasing cache size is easier ;-) – Ukko Feb 14 '17 at 17:30
41

Can a file be saved not sequentially on disk? I mean, part of the file is located under physical address X and the other part under physical address Y which isn't close to X + offset).

Yes; this is known as file fragmentation and is not uncommon, especially with larger files. Most file systems allocate space as it's needed, more or less sequentially, but they can't guess future behaviour — so if you write 200MiB to a file, then add a further 100MiB, there's a non-zero chance that both sets of data will be stored in different areas of the disk (basically, any other write needing more space on disk, occurring after the first write and before the second, could come in between the two). If a filesystem is close to full, the situation will usually be worse: there may not be a contiguous area of free space large enough to hold a new file, so it will have to be fragmented.

Can I somehow control the file sequentiallity? I want to allocate big file of 10GB. I want it to be sequential in disk and not divided between different offsets.

You can tell the filesystem about your file's target size when it's created; this will help the filesystem store it optimally. Many modern filesystems use a technique known as delayed allocation, where the on-disk layout of a new file is calculated as late as possible, to maximise the information available when the calculation is performed. You can help this process by using the posix_fallocate(3) function to tell the filesystem how much disk space should be allocated in total. Modern filesystems will try to perform this allocation sequentially.

Does it act differently between the different types?

Different filesystems behave differently, yes. Log-based filesystems such as NILFS2 don't allocate storage in the same way as extent-based filesystems such as Ext4, and that's just one example of variation.

  • 1
    Will using fallocate(3) ensure file sequentiallity? or will just hint the filesystem? I can't fully understand it from the man pages. – hudac Feb 13 '17 at 13:15
  • 5
    It can't ensure sequential allocation, it's just a hint. But you should definitely use it if you're writing 10GiB files! – Stephen Kitt Feb 13 '17 at 13:31
  • 6
    Essentially all file systems more sophisticated than FAT -- this goes all the way back to the original Berkeley UFS -- will intentionally break up large files and spread them over multiple "allocation groups"; this helps them minimize the overall fragmentation of the disk. There may be a way to adjust how this works, but there's good odds you have to recreate the filesystem from scratch in order to do it, and there probably isn't a way to turn it completely off. – zwol Feb 13 '17 at 14:26
  • 2
    @hudac It's impossible to guarantee sequentiality in all cases (see the case with a drive that is close to being full), and to be honest with the rise of SSDs it matters less than it used to (for those who can afford them at least). – Muzer Feb 13 '17 at 15:02
  • 1
    Also note that there are situations, like RAID systems, where having contiguous files is less efficient, if it's even possible. I think that's really the purpose of a disk/storage subsystem controller: to offload all the work of storing files as optimally as can reasonably be expected. – jamesqf Feb 13 '17 at 18:45
17

The command filefrag will tell you how your file is physically stored on your device:

# filefrag -v /var/log/messages.1 
Filesystem type is: ef53
File size of /var/log/messages.1 is 41733 (11 blocks, blocksize 4096)
 ext logical physical expected length flags
   0       0  2130567               1 
   1       1 15907576  2130568      1 
   2       2 15910400 15907577      1 
   3       3 15902720 15910401      7 
   4      10  2838546 15902727      1 eof
/var/log/messages.1: 5 extents found

If you write your file in one pass, my guess is that your file won't be fragmented.

The man page of fallocate(1) is pretty clear :

fallocate is used to preallocate blocks to a file. For filesystems which support the fallocate system call, this is done quickly by allocating blocks and marking them as uninitialized, requiring no IO to the data blocks. This is much faster than creating a file by filling it with zeros.

As of the Linux Kernel v2.6.31, the fallocate system call is supported by the btrfs, ext4, ocfs2, and xfs filesystems.

Is it sequential? The system will first try to allocate the blocks sequentially. If it can't, it will not warn you.

  • What is type 'ef53'. I saw it also on my files. But my FS type is ext4. – hudac Feb 13 '17 at 16:17
  • 2
    EF53 is the "SUPER_MAGIC" number of ext2, ext3 and ext4. Look in "include/uapi/linux/magic.h" in the kernel sources for all magic numbers of every file-system. – Vouze Feb 13 '17 at 16:34
  • On Debian, filefrag is hidden in /usr/sbin. But it seems to work for ordinary users (on ext4, at least). It may be instructive to strace its operation to see how to measure fragmentation for yourself, if the lack of warning is a hindrance to you. – Toby Speight Feb 14 '17 at 9:30
6

You mention sparse files, and none of the other answers have mentioned them.

Most files are not sparse. The most common way to create a file is to write it all in one go, from the start to the end. No holes there.

However, you are allowed to say "move to position 1,000,000,000,000 and write a byte there." This will create a file that looks like it is an etabyte big, but actually only uses (probably) 4k on disk. This is a sparse file.

You can do this many times for the same file, leaving small amounts of data scattered across the vast emptiness.

While this can be useful, there are two downsides.

The first is that the file will be fragmented, which is what you worried about.

The second is that not all programs handle these files well. E.g. some backup software will try to backup the emptiness and thereby create a backup which is much larger than necessary, possibly too big for the backup medium.

  • But even a non-sparse file will often not be contiguous on disk. – Barmar Feb 15 '17 at 17:43
2

Can I somehow control the file sequentiality? I want to allocate a file of 10GB. I want it to be sequential on disk and not divided between different offsets.

There are at least a couple of ways to achieve this.

  1. Use a filesystem with a lot of spare space and preallocate the space (e.g. use an application specific end-of-data marker and append random data until the filesize reaches 10GB). This isn't guaranteed to result in unfragmented data.

  2. Use a raw (uncooked) filesystem instead of ext4 etc. DBMSs sometimes do this for performance reasons. The tradeoff is you have to do your own caching/journalling/recovery etc if needed.

Instances where you gain much from doing this are relatively rare - I would first look elsewhere to optimise performance.


See also

Is it true that database management systems typically bypass file systems?

-1

If this is just a one-time thing and it's not important how the file is stored originally, just the result matters, then you could just save the file normally and then run the defragmenter of your operating system. Then you can check with this answer if your file is in one piece, if not, repeat. This is the easiest way to do it, without using commands or external programs, but it's certainly not the fastest method, because it defrags your entire disk.

  • 1
    "Run the defragmenter"? Is there such a program? The only thing found when I searched with aptitude search ~ddefrag were ddrescueview and the nids TCP segment reassembly library. Your answer isn't very helpful if you don't say what the program is called, or what arguments need to be passed. – Toby Speight Feb 14 '17 at 16:25
  • 1
    @TobySpeight - yes there is a defragmenter; e4defrag. – ravery Apr 26 '18 at 1:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.