A directory on a typical Unix-like filesystem is just a specially attributed file. So is a symbolic link. Some Unix-like filesystems are able to store a tiny file such as a symbolic blink inside the inode. This means that no block has to be allocated to the file and since file size calculations do not count the inodes, the file has zero size.
A file whose data cannot fit into its inode has to occupy some kind of minimum unit of storage that is separately allocated from the inode. (And then the inode's data space is typically converted into an array of pointers to blocks of that storage.)
The simplest strategy is that everything is divided into indivisible blocks which are all the same size, and no two files can share storage within a single block. However, some filesystems like the Berkeley FFS can break up blocks into fragments assigned to different files in order to reduce waste.
The block size is decided at filesystem creation time, as a compromise between the addressing a huge disk and wasting space for small files. Typical block sizes are powers of two starting at around 512 (probably rare nowadays).
As you know every (non-root) directory contains at least two entries, because even an empty directory contains the . and .. links. If you see nonzero value as the size of even a completely empty directory, that indicates that in the given filesystem, those two basic directory entries are too large to fit into the tiny data area of the inode. So the directory data occupies a separately allocated block, or block fragment.
It looks like on your given filesystem, the block size is 4096 and there are no smaller fragments. (Or else, you have an even larger block size and 4096 is the size of a fragment.) A directory starts occupying a 4096 block/fragment, and when enough entries are added to that directory to run out of space in that unit, the directory grows to 8192, and then to 12 kilobytes and so on.
A large-ish value for .. indicates that the parent is a sizeable directory, which has no bearing on the size of the child directory.