2

Fragmentation seems to create a lot of unnecessary seeks when traversing a directory tree on a HDD:

# stat -c %F 00 01 02
directory
directory
directory

# filefrag -v 00 01 02
Filesystem type is: ef53
File size of 00 is 12288 (3 blocks of 4096 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0..       0:  428351942.. 428351942:      1:            
   1:        1..       2:  428352760.. 428352761:      2:  428351943: last,eof
00: 2 extents found
File size of 01 is 12288 (3 blocks of 4096 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0..       0:  428351771.. 428351771:      1:            
   1:        1..       2:  428891667.. 428891668:      2:  428351772: last,eof
01: 2 extents found
File size of 02 is 12288 (3 blocks of 4096 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0..       0:  428351795.. 428351795:      1:            
   1:        1..       2:  428352705.. 428352706:      2:  428351796: last,eof
02: 2 extents found

e4defrag isn't able to defrag them

# e4defrag -v 00
ext4 defragmentation for directory(00)
[1/116] "00"
    File is not regular file        [ NG ]

So how do I defragment a directory? Not its contents, but the directory itself. The directories are in use, so it should be done atomically, just like defragmenting regular files does not interfere with their use.

Improve this question
6
  • @roaima The title is my question: [How to] atomically defragment ext4 directory files?
    – the8472
    Commented Mar 12, 2017 at 5:21
  • @roaima 1. I said "directory file", referring to the directory object itself (everything is a file, just like block devices are files!) 2. atomically is a word. but "online defragmentation" might work too in this context. 3. I already edited my question.
    – the8472
    Commented Mar 12, 2017 at 9:19
  • "Your example is of a directory." - I intentionally highlighted that with the stat command. So I am quite aware of that.
    – the8472
    Commented Mar 12, 2017 at 9:19
  • You don't, at least not online. Offline, e2fsck has an option to defrag directories, and then there is the old e2defrag. Of course, having only two fragments isn't going to affect performance in any meaningful way.
    – psusi
    Commented Mar 17, 2017 at 13:25
  • @psusi running fsck -D asked to optimize some extent trees (Inode 41947219 extent tree (at level 1) could be shorter. Fix? yes), but it didn't rewrite the directories in question to a single contiguous range.
    – the8472
    Commented Mar 19, 2017 at 21:35

2 Answers 2

Reset to default
2

Since there does not seem to be any online defragmentation tool for directory indices and even the offline defragmenters don't seem to help I had to resort to rebuilding the directory tree recursively.

I've written a small tool (defrag-dirs) for that purpose. Alas, that approach requires the application using the directory tree to be taken down during defragmentation, which can take a considerable amount of time when dealing with millions of files.

Improve this answer
0

If you have the free space, you could create a parallel tree of new directories on the same device and hard link the files to them, then swap names of the head directory, and later after a reboot or other app shutdown, drop the original directory tree. We used to make such clone trees with minimal file space needs to test changes, or to install sw releases beside running production.

You might build each directory directories first (speeding tree traversal), and then oldest files first, as the youngest files are most likely to be ephemeral.

I am not sure how this works on file types other than flat files and directories. I expect symbolic links can themselves be hard linked, and probably named pipes, but devices? And of course I am pretty sure the fake trees in Linux like /prod and /dev, reflecting kernel tables, are off limits.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .