A directory that used to be huge may still have a lot of blocks allocated for directory entries (= names and inode numbers of files and sub-directories in that directory), although almost all of them are now marked as deleted.
When a new directory is created, only a minimum number of spaces are allocated for directory entries. As more and more files are added, new blocks are allocated to hold directory entries as needed. But when files are deleted, the
ext4 filesystem does not consolidate the directory entries and release the now-unnecessary directory metadata blocks, as the assumption is that they might be needed again soon enough.
You might have to unmount the filesystem and run a
e2fsck -C0 -f -D /dev/sda9 on it to optimize the directories, to get the extra directory metadata blocks deallocated and the existing directory entries consolidated to a smaller space.
Since it's your
/home filesystem, you might be able to do it by making sure all regular user accounts are logged out, then logging in locally as root (typically on the text console). If
umount /home in that situation reports that the filesystem is busy, you can use
fuser -m /dev/sda9 to identify the processes blocking you from unmounting
/home. If they are remnants of old user sessions, you can probably just kill them; but if they belong to services, you might want to stop those services in a controlled manner.
The other classic way to do this sort of major maintenance to
/home would be to boot the system into single-user/emergency mode. On distributions using
systemd, the boot option
systemd.unit=emergency.target should do it.
And as others have mentioned, there is an even simpler solution, if preserving the timestamps of the directory is not important, and the problem directory is not the root directory of the filesystem it's in: create a new directory alongside the "bloated" one, move all files to the new directory, remove the old directory, and rename the new directory to have the same name as the old one did. For example, if
/directory/A is the one with the problem:
mv /directory/A/* /directory/B/ # regular files and sub-directories
mv /directory/A/.??* /directory/B/ # hidden files/dirs too
mv /directory/B /directory/A
Of course, if the directory is being used by any services, it would be a good idea to stop those services first.