As Fox noted in their answer, the kernel handles the
open() syscall, and meanwhile, filesystems implement file operations in their own way. Filesystems, on the other hand, implement their version of syscalls, and that's what kernel should be using.
Consider, for instance, ext4 call to open directory or file operations in procfs (which notably has no
.open mentioned anywhere), and pipefs which handles named and unnamed pipes. But the general idea is that the
open() syscall is not going to be necessarily assembly, nor it is guaranteed to be specific to a particular architecture.
And to quote an answer by the user Warren Young who noted that way before this question appeared,
there is no single file where mkdir() exists. Linux supports many different file systems and each one has its own implementation of the "mkdir" operation. The abstraction layer that lets the kernel hide all that behind a single system call is called the VFS. So, you probably want to start digging in fs/namei.c, with vfs_mkdir(). The actual implementations of the low-level file system modifying code are elsewhere. For instance, the ext4 implementation is called ext4_mkdir(), defined in fs/ext4/namei.c.
As for why
open() works this way, this is also due to Unix design philosophy that everything is a file. If you're using an API, you want to deal with consistent interface and not reinvent the wheel for every filesystem in existence (unless a person is a kernel developer, in which case they have our gratitude and respect).