Sorry for repeating but take a look to
git clone git://microwindows.org/microwin
In particular take a look to the files:
What is done in the tty driver is very similar to what Xorg does, and The
devfb driver is a very simple and clean implementation.
Linux's devfb frame buffers mostly rely on
ioctl (eg: to set/get resolution) and
mmap (to raw write/read pixels).
devfb is just one (easy and a bit more portable on linux) way to access the graphic hardware.
Xorg drivers instead are composed by a kernel driver and a Xorg user space interface between the driver and Xorg itself, and what happen between kernel and user side is really implementation-dependant (there isn't a standard).
You can also take a look to
Directfb but Nano-X is the cleanest/easy and a display server itself, so probably could help you on other question that you'll surely meet.