You seem to be confused about what gdm and x11 do.
The X server uses the X11 protocol to allow applications to use graphical hardware and input devices. It allows them to draw windows on the screen, perform certain operations on those windows, react to user input events, and close the windows again. GDM is one such application. As far as the X server is concerned, it is not a special application. It allows users to log on, but the X server has no concept of users and passwords, it just knows about applications.
It is perfectly possible to use the X server without GDM or another display manager. If you do that, you still need to configure the X server, and doing that is done with the same programs as when you do use GDM (i.e., xrandr and others). Alternatively, you may also be using the framebuffer, in which case configuration is done through other means. However, as this is somewhat more involved and not as efficient (since the X server has better drivers for'most display hardware than the kernel framebuffer), I would recommend against it.