Decrease time to shrink ext4 filesystem

Question

I need to shrink a large ext4 volume, and I would like to do it with as little downtime as possible. With the testing I've done so far it looks like it could be unmounted for the resize for up to a week. Is there any way to defragment the filesystem online ahead of time so that resizefs won't have to move so many blocks around?

Update: It's taken some time to get to this point, moved quite a few TB of data around in preparation for the shrink, and I've been experimenting using the information in the answer below. I finally came up with the following command-line which could be useful to others in a similar situation with only minor modifications. Also note, it should be run as root for the filefrag and e4defrag commands to work properly - it won't affect the file ownership. It does also work properly on files with multiple hard-links, which I have lots of.

find -type f -print0 | xargs -0 filefrag -v | grep '\.\.[34][0-9]\{9\}.*eof' -A 1 | awk '/extents found/ {match($0, /^(.*): [0-9]+ extents found/, res); print res[1]}' | xargs -n 1 -d '\n' e4defrag

A quick explanation to make it easier for others to modify/use:

The first 'find' command builds the list of files to work with. Possibly redundant now or could be done a better way, but while testing I had other filters there and I've left it as a handy place to modify the scope of the rest of the command.

Next pass each file through 'filefrag -v' to get a list of all physical blocks used by each file.

The grep looks for the last block used by each file (line ending in 'eof'), and where that block is a 10-digit number starting with 3 or 4. In my case my new filesystem size will be 2980024320 blocks long so that does a good-enough job of only working on files that are on the area of disk to be removed. Having grep also include the following line (the '-A 1') also includes the filename in the output for the next section. This is where anyone else doing this will have to modify the command depending on the size of their filesystem. It could also probably be done in a much better way but this is working for me now and I'm lazy.

awk pulls just the filenames out of all the other garbage that grep left in the filefrag output.

And finally e4defrag is called - I don't care about the actual fragment count, but it has the side effect of moving the physical blocks around (hopefully into an early part of the drive), and it works against files with multiple hard-links with no extra effort.

If you only want to know which files it would defrag without actually moving any data around, just leave the last piece of the command off.

find -type f -print0 | xargs -0 filefrag -v | grep '\.\.[34][0-9]\{9\}.*eof' -A 1 | awk '/extents found/ {match($0, /^(.*): [0-9]+ extents found/, res); print res[1]}'

Answer 1

From what I can tell, ext4fs supports online defragmentation (it's listed under "done", but the status field is empty; the original patch is from late 2006) through e4defrag in e2fsprogs 1.42 or newer which when running on Linux 2.6.28 or newer allows you to query status for directories or possibly file systems, and at least defragment individual files. e2fsprogs as of today is at version 1.42.8.

I'm not sure whether or not this helps you, though, as what you want to do doesn't seem to be so much defragment the data as consolidate the data on disk. The two are often done together, but they are distinctly different operations.

A simple way to consolidate the data, which might work, assuming you have a reasonable amount of free space, is to copy each file to some other logical logication on the same file system, and then use mv to replace the data pointed to by the inode with the new copy. It would depend heavily on exactly how the ext4 allocator works in detail, but it might be worth an attempt and it should be fairly easy to script. Just watch out for files that are hardlinked from more than one place (with a scheme like this it might be easiest to simply ignore any files with link count > 1, and let resizefs deal with those).