The X11 server does not need any old terminal device. It specifically needs a kernel virtual terminal device.
This is because it isn't using the terminal for general terminal I/O. It is using the terminal as a way of negotiating sharing I/O devices between it and the terminal emulator program that is built into the kernel. Using a protocol that is specific to kernel virtual terminal devices, which involves
ioctl()s that only that type of device understands and sending signals to processes, it can negotiate when it is in charge of the display and Human Input Devices (HIDs) and when the kernel built-in terminal emulator program is.
(On Linux, this is the primary use of the KVT, with I/O routed through other devices entirely, and it being the controlling terminal is almost, but not quite, a side effect of fact that the X11 server opens the KVT device in a session that has no controlling terminal. On the BSDs, things are more complex, as the KVT device is also a way of accessing the framebuffer and HID input, albeit not as good a one as accessing the underlying USB HIDs if they are USB. However, on FreeBSD at least
Xorg does not end up with the KVT as its controlling terminal in the first place, because on FreeBSD simply opening a terminal device never automatically makes it one's controlling terminal, and the question is moot because the
Xorg process actually does not have a controlling terminal in normal operation.)
If the kernel's built-in terminal emulator is not using a particular display device or HID, which can be arranged with some effort (in ways that are well beyond the scope of this answer), then the device sharing is unnecessary and the X11 server can just "own" the device(s) all of the time. Moreover, if the kernel has been built without the built-in terminal emulator, there is nothing to negotiate with. Similarly, if an X11 server isn't actually using any I/O devices in the first place (as is the case with the
Xvnc program) a mechanism for negotiating the ownership of I/O devices is unnecessary.