Why does "ls" take extremely long in a small directory that used to be big? How to fix this?

Question

I'm running Arch Linux, and use ext4 filesystems.

When I run ls in a directory that is actually small now, but used to be huge - it hangs for a while. But the next time I run it, it's almost instantaneous.

I tried doing:

strace ls

but I honestly don't know how to debug the output. I can post it if necessary, though it's more than a 100 lines long.

And, no, I'm not using any aliases.

$ type ls
ls is hashed (/usr/bin/ls)

$ df .
Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/sda9      209460908 60427980 138323220  31% /home

Answer 1

A directory that used to be huge may still have a lot of blocks allocated for directory entries (= names and inode numbers of files and sub-directories in that directory), although almost all of them are now marked as deleted.

When a new directory is created, only a minimum number of spaces are allocated for directory entries. As more and more files are added, new blocks are allocated to hold directory entries as needed. But when files are deleted, the ext4 filesystem does not consolidate the directory entries and release the now-unnecessary directory metadata blocks, as the assumption is that they might be needed again soon enough.

You might have to unmount the filesystem and run a e2fsck -C0 -f -D /dev/sda9 on it to optimize the directories, to get the extra directory metadata blocks deallocated and the existing directory entries consolidated to a smaller space.

Since it's your /home filesystem, you might be able to do it by making sure all regular user accounts are logged out, then logging in locally as root (typically on the text console). If umount /home in that situation reports that the filesystem is busy, you can use fuser -m /dev/sda9 to identify the processes blocking you from unmounting /home. If they are remnants of old user sessions, you can probably just kill them; but if they belong to services, you might want to stop those services in a controlled manner.

The other classic way to do this sort of major maintenance to /home would be to boot the system into single-user/emergency mode. On distributions using systemd, the boot option systemd.unit=emergency.target should do it.

And as others have mentioned, there is an even simpler solution, if preserving the timestamps of the directory is not important, and the problem directory is not the root directory of the filesystem it's in: create a new directory alongside the "bloated" one, move all files to the new directory, remove the old directory, and rename the new directory to have the same name as the old one did. For example, if /directory/A is the one with the problem:

mkdir /directory/B
mv /directory/A/* /directory/B/      # regular files and sub-directories
mv /directory/A/.??* /directory/B/   # hidden files/dirs too
rmdir /directory/A
mv /directory/B /directory/A

Of course, if the directory is being used by any services, it would be a good idea to stop those services first.

Answer 2

Out of curiosity, let's try to reproduce this:

$ mkdir test
$ cd test
$ time ls   # Check initial speed of ls
real    0m0,002s
$ stat .    # Check initial size of directory
  File: .
  Size: 4096        Blocks: 8          IO Block: 4096   directory
  ...
$ seq 1 1000000 | xargs touch    # Create lot of files
$ echo 3 | sudo tee /proc/sys/vm/drop_caches   # Clear cache
$ time ls > /dev/null
real    0m1.588s
$ stat .                        # Check size of directory when files are there
  File: .
  Size: 22925312    Blocks: 44776      IO Block: 4096   directory

Ok, so now we have a large directory. Let's remove the files and see what happens:

$ ls | xargs rm   # To avoid too long argument list
$ echo 3 | sudo tee /proc/sys/vm/drop_caches
$ time ls > /dev/null
real    0m1.242s
$ stat .
 File: .
 Size: 22925312     Blocks: 44776      IO Block: 4096   directory

So yes, the allocated size for the directory does stay large and that does cause slow ls, like telcoM's answer also indicated.

If it is just a single directory with the problem, there is a simpler solution that does not require unmounting or root access: Simply create a new directory, move remaining files to it and remove the bloated one.