You can't, not on ext4 anyway.
ext4 never shrinks a directory's size (i.e. the size taken by the directory itself, not its contents) once it has grown. AFAIK, this is true on several (most?) other fs types, but not all fs types. This can seriously impact performance on some filesystems where a directory has had many thousands or millions of files in it, even if those files have been deleted or moved.
BTW, not all filesystems suffer serious performance problems once a directory has had lots of files in it (IIRC, it isn't a problem on xfs, and is less of a problem on ext4 than it used to be with ext3 or ext2).
The solution is to move the contents to a new directory with the same name. For example:
mv dir dir.old
chmod --reference=dir.old/ dir/
chown --reference=dir.old/ dir/
mv dir.old/* dir/
chown ensure that the new directory has exactly the same owner, group, and permissions as the old. If you're using ACLs, you'll have to copy those too.
Note: this won't work if the directory is actually a mount-point for a filesystem. AFAIK, the only way to shrink the top-level dir of a filesystem is to backup, reformat with mkfs, and restore.
Update: the second duplicate link used to close this question says that you can
umount the fs and use
e2fsck -C0 -f -D /dev/XXX. See
man e2fsck for details on what these options do.
Also note: whether using
mv, backup-reformat-restore, or
fsck, you should do this when the directory and the files in it are not in use by any process. Stop any processes using files in the directory. If necessary, reboot to single-user/emergency mode.