The origins and design features are different. It's generally a better idea to design a protocol from scratch because then you can make it specifically suitable for what you need, without having problems with legacy protocols.
I believe that this choice was because the in-kernel implementation of UNIX domain sockets is too different to allow sane porting to user-kernel communication. The list of protocols for AF_NETLINK is specifically designed to provide ioctl-like control. AF_UNIX doesn't even use the "protocol" argument to the
socket function. And if they somehow extended the definition to allow additional protocols, it would probably break existing applications, and it would be very awkward in-kernel to redirect the new protocols to kernel control code. It may also be a security risk to overlap these two functionalities (especially since AF_UNIX was not originally designed with this in mind).
And lastly... UNIX domain sockets use filesystem as a namespace (although there is a hack that allows "anonymous" sockets). As such, they are immediately available to all users that have permissions to this socket -- to communicate with the kernel, there would need to be an always-open "file" somewhere in the filesystem (sys probably?) that users would use to connect to the kernel.
In short, they are simply meant to be used for different things. Even if you can reuse AF_NETLINK for two userspace processes (it's done differently than in AF_UNIX) the opposite wouldn't really work.