To quote Terry Wang's answer:
A directory is just a special file which contains an array of filenames and inode numbers. When the directory was created, the file system allocated 1 inode to the directory with a "filename" (dir name in fact). The inode points to a single data block (minimum overhead), which is 4096 bytes.
Directories can grow larger than 4096, and one of the peculiar "bugs" of ext4 filesystem which did not receive much priority yet is that after directory information has outgrown 4096 bytes, that number is not decremented if files are removed. And probably the big news flash is that how information is allocated for a directory depends on the filesystem and even the options which are enabled/disabled( source ), so the special number 4096 is specific to ext4 in this case, but could be something else on other filesystems such as UFS for example.
Note, however, Terry's talking about inode which represents directory, directory itself being a dirent in parent directory. Initial directory contents are dirents or links
The ext4 filesystem defaults to creating directories ( even if empty ) with 4096 bytes allocation. That's the key. According to ServerFault and Arch Linux forum when directory shows as 0 in size it means the directory is located or hosts a filesystem other than ext4.
And this is true of virtual filesystems (which effectively aren't physical on-disk filesystems, but rather are an interface exposed by kernel and are contained in memory):
$ ls -ld /proc /sys
dr-xr-xr-x 243 root root 0 Feb 24 13:57 /proc
dr-xr-xr-x 13 root root 0 Feb 26 14:42 /sys