44

I'm running Arch Linux, and use ext4 filesystems.

When I run ls in a directory that is actually small now, but used to be huge - it hangs for a while. But the next time I run it, it's almost instantaneous.

I tried doing:

strace ls

but I honestly don't know how to debug the output. I can post it if necessary, though it's more than a 100 lines long.

And, no, I'm not using any aliases.

$ type ls
ls is hashed (/usr/bin/ls)

$ df .
Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/sda9      209460908 60427980 138323220  31% /home
15
  • How many entries does the directory have?
    – Cyrus
    Apr 24, 2021 at 22:46
  • 2
    72 entries. I use Arch.
    – Belen
    Apr 24, 2021 at 22:47
  • 1
    Is ls -f much faster? Show output of df . in this directory.
    – Cyrus
    Apr 24, 2021 at 22:48
  • 3
  • 7
    @jamesqf The time it takes to retrieve directory entries is directly proportional to the total number of entries to retrieve (though it may be quantized to some extent). It’s unusual with fast storage because most directories do not have multiple thousands of entries, but historically this was a bigger issue. Even today though it can still happen, at my last job I had to deal with a lack of a proper folder structure on a major fileserver, which resulted in a directory with more than 70000 entries, which would take about 30 seconds to run ls on despite the server having very fast storage. Apr 25, 2021 at 22:20

2 Answers 2

62

A directory that used to be huge may still have a lot of blocks allocated for directory entries (= names and inode numbers of files and sub-directories in that directory), although almost all of them are now marked as deleted.

When a new directory is created, only a minimum number of spaces are allocated for directory entries. As more and more files are added, new blocks are allocated to hold directory entries as needed. But when files are deleted, the ext4 filesystem does not consolidate the directory entries and release the now-unnecessary directory metadata blocks, as the assumption is that they might be needed again soon enough.

You might have to unmount the filesystem and run a e2fsck -C0 -f -D /dev/sda9 on it to optimize the directories, to get the extra directory metadata blocks deallocated and the existing directory entries consolidated to a smaller space.

Since it's your /home filesystem, you might be able to do it by making sure all regular user accounts are logged out, then logging in locally as root (typically on the text console). If umount /home in that situation reports that the filesystem is busy, you can use fuser -m /dev/sda9 to identify the processes blocking you from unmounting /home. If they are remnants of old user sessions, you can probably just kill them; but if they belong to services, you might want to stop those services in a controlled manner.

The other classic way to do this sort of major maintenance to /home would be to boot the system into single-user/emergency mode. On distributions using systemd, the boot option systemd.unit=emergency.target should do it.

And as others have mentioned, there is an even simpler solution, if preserving the timestamps of the directory is not important, and the problem directory is not the root directory of the filesystem it's in: create a new directory alongside the "bloated" one, move all files to the new directory, remove the old directory, and rename the new directory to have the same name as the old one did. For example, if /directory/A is the one with the problem:

mkdir /directory/B
mv /directory/A/* /directory/B/      # regular files and sub-directories
mv /directory/A/.??* /directory/B/   # hidden files/dirs too
rmdir /directory/A
mv /directory/B /directory/A

Of course, if the directory is being used by any services, it would be a good idea to stop those services first.

8
  • 3
    Just fair warning, any systemd service that has ProtectHome enabled will have /home in a private namespace and you won’t be able to unmount /home, and it won’t show up in fuser, because it is a kernel mount. I believe CUPS is one of those services by default.
    – jsbillings
    Apr 25, 2021 at 2:02
  • 1
    This worked. Though I had to add the -f flag because it was otherwise reporting that the system was clean without actually checking it.
    – Belen
    Apr 25, 2021 at 8:38
  • 3
    @Belen Thanks for the feedback, I edited the -f flag into the answer. It's been quite a while since I had to do this, so I didn't remember that detail.
    – telcoM
    Apr 25, 2021 at 9:20
  • 2
    @OrangeDog - cp -a should be more efficient than rsync as no contents will need to be copied (just new directory entries in the target created). Unless there is something cp -a does not preserve that rsync does? Apr 26, 2021 at 16:16
  • 2
    Just an important note: the correct "fix" here in nearly all conceivable instances is creating a new directory, moving all the files into it, and then deleting the old directory. It's very unlikely that you'd need to fsck. The mv(1) command preserves timestamps. By design, one should not do any file activity in the root of any mounted filesystem - always do it in a subdirectory. With the exception of /tmp, where it's considered reasonable to remove all files on a reboot.
    – Brian C
    Apr 28, 2021 at 2:31
44

Out of curiosity, let's try to reproduce this:

$ mkdir test
$ cd test
$ time ls   # Check initial speed of ls
real    0m0,002s
$ stat .    # Check initial size of directory
  File: .
  Size: 4096        Blocks: 8          IO Block: 4096   directory
  ...
$ seq 1 1000000 | xargs touch    # Create lot of files
$ echo 3 | sudo tee /proc/sys/vm/drop_caches   # Clear cache
$ time ls > /dev/null
real    0m1.588s
$ stat .                        # Check size of directory when files are there
  File: .
  Size: 22925312    Blocks: 44776      IO Block: 4096   directory

Ok, so now we have a large directory. Let's remove the files and see what happens:

$ ls | xargs rm   # To avoid too long argument list
$ echo 3 | sudo tee /proc/sys/vm/drop_caches
$ time ls > /dev/null
real    0m1.242s
$ stat .
 File: .
 Size: 22925312     Blocks: 44776      IO Block: 4096   directory

So yes, the allocated size for the directory does stay large and that does cause slow ls, like telcoM's answer also indicated.

If it is just a single directory with the problem, there is a simpler solution that does not require unmounting or root access: Simply create a new directory, move remaining files to it and remove the bloated one.

12
  • 4
    e2fsck defrag/optimize has the advantage of preserving timestamps including ctime, but yes it's certainly simpler to just mkdir and mv, unless it's the root directory of a filesystem. Apr 25, 2021 at 13:10
  • 1
    I thought of that, but I wanted to preserve the name. (Creating a new directory, moving everthing there, and renaming that directory with mv didn't solve the problem.) Besides, I'm glad I asked because I learned why that problem was ocurring.
    – Belen
    Apr 25, 2021 at 15:44
  • 2
    @Belen That's quite surprising, as the directory entry should be linked to the inode and not the name.
    – jpa
    Apr 25, 2021 at 15:48
  • 5
    @Belen If you do mkdir B; mv A/* B/; mv B Ayou'll end up with the files in A/B/* and the A directory still bloated. The rmdir A step before the second mv is important in this case.
    – telcoM
    Apr 25, 2021 at 16:30
  • Now that you mention it, you're right, I didn't do rmdir A.
    – Belen
    Apr 25, 2021 at 17:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.