The term "graphics driver" is used to refer to several different things. One of them is a kernel driver. The kernel driver mostly just sets the video mode and facilitates passing data to/from the card. It also usually downloads the firmware into the GPU on the card. The firmware is a program that the GPU itself runs, but unfortunately, graphics vendors only provide it as a binary blob so you can't look at its source code.
Above that you usually have Xorg running, which has its own driver that translates generic X11 or OpenGL drawing calls into commands the card understands, and sends them down to the card to execute. It also may do some of the work itself depending on what commands the gpu does and does not support. In the case of the OpenGL calls, the Direct Rendering Infrastructure allows this part of the driver to actually execute directly in the client application rather than the X server, in order to get acceptable performance. It also allows the driver in the client application to send its commands directly to the gpu, thanks to coordination with and help from Xorg and the kernel driver at startup.
Wayland and Mir are supposed to replace Xorg as a simplified type of display server.
Unity is both a shell ( provides desktop/launcher ) and compositing window manager in one.
GNOME and KDE are desktop environments. They are large projects consisting of many components. The core of them are their respective application toolkits, which are GTK for GNOME and Qt for KDE. This is a library framework that an application is written with and provides the foundation on which everything else is built. Some of the basic services they provide are event and object handling, Windows, basic drawing functions, I/O, and much more.