19

I have a 900GB ext4 partition on a (magnetic) hard drive that has no defects and no bad sectors. The partition is completely empty except for an empty lost+found directory. The partition was formatted using default parameters except that I set the number of reserved filesystem blocks to 1%.

I downloaded the ~900MB file xubuntu-15.04-desktop-amd64.iso to the partition's mount point directory using wget. When the download was finished, I found that the file was split into four fragments:

filefrag -v /media/emma/red/xubuntu-15.04-desktop-amd64.iso
Filesystem type is: ef53
File size of /media/emma/red/xubuntu-15.04-desktop-amd64.iso is 1009778688 (246528 blocks of 4096 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0..   32767:      34816..     67583:  32768:            
   1:    32768..   63487:      67584..     98303:  30720:            
   2:    63488..   96255:     100352..    133119:  32768:      98304:
   3:    96256..  126975:     133120..    163839:  30720:            
   4:   126976..  159743:     165888..    198655:  32768:     163840:
   5:   159744..  190463:     198656..    229375:  30720:            
   6:   190464..  223231:     231424..    264191:  32768:     229376:
   7:   223232..  246527:     264192..    287487:  23296:             eof
/media/emma/red/xubuntu-15.04-desktop-amd64.iso: 4 extents found

Thinking this might be releated to wget somehow, I removed the ISO file from the partition, making it empty again, then I copied the ~700MB file v1.mp4 to the partition using cp. This file was fragmented too. It was split into three fragments:

filefrag -v /media/emma/red/v1.mp4
Filesystem type is: ef53
File size of /media/emma/red/v1.mp4 is 737904458 (180153 blocks of 4096 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0..   32767:      34816..     67583:  32768:            
   1:    32768..   63487:      67584..     98303:  30720:            
   2:    63488..   96255:     100352..    133119:  32768:      98304:
   3:    96256..  126975:     133120..    163839:  30720:            
   4:   126976..  159743:     165888..    198655:  32768:     163840:
   5:   159744..  180152:     198656..    219064:  20409:             eof
/media/emma/red/v1.mp4: 3 extents found

Why is this happening? And is there a way to prevent it from happening? I thought ext4 was meant to be resistant to fragmentation. Instead I find that it immediately fragments a solitary file when all the rest of the volume is unused. This seems to be worse than both FAT32 and NTFS.

  • 4
    I'm trying to imagine under what circumstances this could possibly matter, and I'm coming up empty. – Greg Hewgill May 18 '15 at 1:53
  • 4
    @GregHewgill: It mattered because I thought it was abnormal. Now I know that it's normal, it doesn't matter. – EmmaV May 18 '15 at 3:00
17

3 or 4 fragments in a 900mb file is very good. Fragmentation becomes a problem when a file of that size has more like 100+ fragments. It isn't uncommon for fat or ntfs to fragment such a file into several hundred pieces.

You generally won't see better than that at least on older ext4 filesystems because the maximum size of a block group is 128 MB, and so every 128 MB the contiguous space is broken by a few blocks for the allocation bitmaps and inode tables for the next block group. A more recent ext4 feature called flex_bg allows packing a number of ( typically 16 ) block groups' worth of these tables together, leaving longer runs of allocatable blocks but depending on your distribution and what version of e2fsprogs was used to format it, this option may not have been used.

You can use tune2fs -l to check the features enabled when your filesystem was formatted.

  • Very interesting. I assumed all the inode tables etc. were at the start of the volume. – EmmaV May 18 '15 at 3:02
  • 1
    @EmmaV distributing them across the disk, relatively close to the data they refer to, results in shorter seeks and faster disk access :) – hobbs May 18 '15 at 16:22
10

I can't truly answer but I think this might help:

Notice how each fragment is, at most, 32768 blocks in size (a power of 2, that should raise a flag that something is going on, and also give you a hint for something to look for).

Also worth noting, those physical offsets between extents are pretty close to each other.

From: Ext4 Disk Layout

An ext4 file system is split into a series of block groups. To reduce performance difficulties due to fragmentation, the block allocator tries very hard to keep each file's blocks within the same group, thereby reducing seek times. The size of a block group is specified in sb.s_blocks_per_group blocks, though it can also calculated as 8 * block_size_in_bytes. With the default block size of 4KiB, each group will contain 32,768 blocks, for a length of 128MiB

And further down:

The first tool that ext4 uses to combat fragmentation is the multi-block allocator. When a file is first created, the block allocator speculatively allocates 8KiB of disk space to the file [...] A second related trick that ext4 uses is delayed allocation. Under this scheme, when a file needs more blocks to absorb file writes, the filesystem defers deciding the exact placement on the disk until all the dirty buffers are being written out to disk. By not committing to a particular placement until it's absolutely necessary (the commit timeout is hit, or sync() is called, or the kernel runs out of memory), the hope is that the filesystem can make better location decisions.

So I'd say the allocator only cares about data locality within the block group (those 32K blocks), but not about block groups being contiguous to each other.

  • The first quote you gave answers my question. – EmmaV May 18 '15 at 3:06
  • 1
    Each extent has a maximum of 32k blocks because that is the maximum length an extent descriptor can cover. Extents are not fragments. If you notice several of the extents' physical blocks immediately follow those of the previous extent, and so do not constitute a fragment ( 6 extents vs 3 fragments ). – psusi May 18 '15 at 13:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.