1

The way I understand it, with the Linux ext3 filesystem at least, when a large file is created, the blocks are not actually allocated on to the disk until data is actually written to that portion of the file. Linux does a similar thing when programs allocate memory, where the allocations are lazy and pages aren't actually allocated until data is written to them, but that's another topic.

When file sharing programs like Bittorrent and Gnutella write to files, they write small blocks in random areas of the file. The Linux filesystem places these portions of the file all over on the disk creating a lot of fragmentation. When writing to several files at once like this, the problem is especially bad. Even selecting preallocate files in the file sharing program doesn't seem to help. Being able to read these files at only 10MB/sec is typical with the amount of fragmentation that results. The filefrag utility can be used to see how many fragments a file has. A file like /var/log/messsages is a good place to test.

With ext3, and I assume most of the other Linux filesystems as well, some data actually has to be written to the entire file in order from beginning to end to actually have a fully allocated file with minimal fragmentation.

I could try mounting an NTFS volume in Linux, but there has to be a better way! There is ext4, btrfs, xfs, jfs, ReiserFS, ExFAT, and even FAT32 to consider! Do any of these preallocate files, or at least have an API that allows doing this without actually having to write data to the entire file first?

1

1 Answer 1

1

In 2024 the following Linux filesystems support file preallocation, posix_fallocate():

Local filesystems:

  • bcachefs (highly experimental at the moment, not recommended for general use)
  • btrfs
  • ext4/ext3
  • F2FS
  • NTFS3 (a prealloc mount option must be specified to enable the feature)
  • VFAT/FAT32
  • XFS

Network filesystems:

  • GFS2
  • NFS
  • OCFS2
  • CIFS/SAMBA

Important considerations:

  • The application must make use of this feature
  • Preallocation may not yield optimal results if your filesystem is already [highly] fragmented and there are no sufficiently sized holes.
  • Given there's enough free space to align files properly:
    • Under Linux only XFS can be fully defragmented (meaning you can defragment free space as well)
    • In Windows you can obviously fully defragment NTFS/VFAT.
2
  • How do you determine if the disk space is allocated, or if actual storage areas on the FS are assigned (which would reduce fragmentation)? The manual page for posix_fsallocate() says "After a successful call to posix_fsallocate(), subsequent writes to bytes in the specified range are guaranteed not to fail because of lack of disk space." Commented Apr 21 at 23:52
  • Linux doesn't provide APIs for that. I remember there was an app which showed ext2/ext3/ext4 file allocation bitmap graphically but that's it. Found it! github.com/i-rinat/fragview Commented Apr 22 at 5:08

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .