How is the linux graphics stack organised?

Question

Can anybody explain(hopefully with a picture), how is the linux graphics stack organised? I hear all the time about X/GTK/GNOME/KDE etc., but I really don't have any idea what they actually do and how they interact with each other and other portions of the stack. How do Unity and Wayland fit in?

Answer 1

The X Window System uses a client-server architecture. The X server runs on the machine that has the display (monitors + input devices), while X clients can run on any other machine, and connect to the X server using the X protocol (not directly, but rather by using a library, like Xlib, or the more modern non-blocking event-driven XCB). The X protocol is designed to be extensible, and has many extensions (see xdpyinfo(1)).

The X server does only low level operations, like creating and destroying windows, doing drawing operations (nowadays most drawing is done on the client and sent as an image to the server), sending events to windows, ... You can see how little an X server does by running X :1 & (use any number not already used by another X server) or Xephyr :1 & (Xephyr runs an X server embedded on your current X server) and then running xterm -display :1 & and switching to the new X server (you may need to setup X authorization using xauth(1)).

As you can see, the X server does very little, it doesn't draw title bars, doesn't do window minimization/iconification, doesn't manage window placement... Of course, you can control window placement manually running a command like xterm -geometry -0-0, but you will usually have an special X client doing the above things. This client is called a window manager. There can only be one window manager active at a time. If you still have open the bare X server of the previous commands, you can try to run a window manager on it, like twm, metacity, kwin, compiz, larswm, pawm, ...

As we said, X only does low level operations, and doesn't provide higher level concepts as pushbuttons, menus, toolbars, ... These are provided by libraries called toolkits, e.g: Xaw, GTK, Qt, FLTK, ...

Desktop environments are collections of programs designed to provide a unified user experience. So desktop environments typically provides panels, application launchers, system trays, control panels, configuration infrastructure (where to save settings). Some well known desktop environments are KDE (built using the Qt toolkit), Gnome (using GTK), Enlightenment (using its own toolkit libraries), ...

Some modern desktop effects are best done using 3d hardware. So a new component appears, the composite manager. An X extension, the XComposite extension, sends window contents to the composite manager. The composite manager converts those contents to textures and uses 3d hardware via OpenGL to compose them in many ways (alpha blending, 3d projections, ...).

Not so long ago, the X server talked directly to hardware devices. A significant portion of this device handling has been moving to the OS kernel: DRI (permitting access to 3d hardware by X and direct rendering clients), evdev (unified interface for input device handling), KMS (moving graphics mode setting to the kernel), GEM/TTM (texture memory management).

So, with the complexity of device handling now mostly outside of X, it became easier to experiment with simplified window systems. Wayland is a window system based on the composite manager concept, i.e. the window system is the composite manager. Wayland makes use of the device handling that has moved out of X and renders using OpenGL.

As for Unity, it's a desktop environment designed to have a user interface suitable for netbooks.

Answer 2

The traditional stack is built upon 3 main components:

• X server which handles displaying
• Window manager that put windows into frames, handles minimizing windows etc. That's part of separation of mechanism from policy in Unix
• Clients that perform useful task as displaying stackexchange website. They may use X protocol directly (suicide), use xlib or xcb (slightly easier) or use toolkit such as GTK+ or QT.

The X architecture was made network hence allowing the clients to be on separate host then server.

So far so good. However that was the image from way back. Nowadays it is not the CPU that handles the graphics but the GPU. There have been various attempts to incorporate it in the model - and place when kernel come into place to greater extend.

Firstly there have been made some assumptions regarding the usage of graphic card. For example only the on-screen rendering was assumed. I cannot find this information on wikipedia right now but DRI 1 also assumed that only one application will use OpenGL at the same time (I cannot quote right away but I can dig the bug up where it was close as WONTFIX with note to wait to DRI 2).

A few temporary solution have been proposed for indirect rendering (needed for composite WM):

• XGL - early proposition supporting applications talking directly to card
• AIGLX - accepted proposition that uses network properties of OpenGL protocol
• Proprietary solution of NVidia

The works on newer architecture (DRI 2) was started. That included:

• In-kernel support for memory handling (GEM/TTM)
• Kernel modesetting (KMS) allowing to change resolution in-kernel hence avoiding delays when switching between X and console & few other features (like displaying message on panic even if X are running - which IIRC is planned to be implemented).

Somehow orthogonal to move to kernel the work on Gallium drivers have been started. Mesa library started as implementation of OpenGL on CPU and then it started using GPU acceleration. It always have been tighten to OpenGL. In OpenGL 3.0 the model have changed significantly and the rewrite of library was inevitable. However they are taking the opportunity to split the code into several layer extracting common code and providing low-level API allowing to implement various 3D API on top of it - allowing for example Wine to provide DirectX talking directly to Gallium instead of going through the OpenGL API layer (which may not have direct 1-1 calls).

---

Wayland is a projects which considers the above a bit complicated and with too "history". The design from 1984 (although highly modified and adapted) does not corresponds to beginning of 21 c. according to proponents.

It is suppose to offer greater flexibility and better performance although it is still missing some important features like full OpenGL support (and important for some - network support).

---

A bit more about desktop environments and window managers. The window manager is an application responsible for how window will behave - for example it is responsible for managing workspaces, drawing titlebar(the thing on top of screen with windo title and minimise/maximise/close buttons) etc.

Firstly only minimal WM was used but afterwards the user started wanting desktop environments - i.e. more featured version, which included menu start, desktop background etc. However most parts of the desktop environment is not a window manager although they are often integrated.

After some time the composite WM were introduced which uses OpenGL and indirect rendering to do their work. They provides not only graphical effects but also allow easier implementation of some accessibility features (like magnifier).

Answer 3

First of all, there is really no Linux graphics stack. Linux has no graphical display capabilities.

However, Linux applications can use graphical displays and there are a number of different systems for doing that. The most common ones are all built on top of X windows.

X is a network protocol because in the middle of an X protocol stack you can have a network as a standard component. Let's look at a specific use case. A physicist in Berlin wants to run an experiment at the CERN in Switzerland on one of the nuclear particle colliders. He logs in remotely and runs a data analysis program on one of CERN's supercomputer arrays and graphs the results on his screen.

In Berlin, the physicist has an X-terminal device running some X-server software that is providing a graphical display capability to remote applications. The X-server software has a framebuffer that talks to the specific device driver for the specific hardware. And the X-server software speaks the X protocol. So the layers might be graphical device->device driver->frame buffer->X server->X protocol.

Then, in Switzerland, the application connects to a display using the X protocol and sends graphic display commands like "draw rectangle" or "alpha blend". The application probably uses a high level graphical library and that library will likely, in turn, be based on a lower level library. For instance the application may be written in Python using the WxWidget toolkit which is built on top of GTK which uses a library called Cairo for core graphical drawing commands. There may also be OPENGL also on top of Cairo. The layers might be like this: WxWidgets->GTK->Cairo->X Toolkit->X protocol. Clearly it is the protocol in the middle that connects things, and since Linux also supports UNIX sockets, a completely internal transport for data, these two types of things can run on one machine if you want to.

X refers to the protocol and anything fundamental to the architecture such as the X-server that runs the graphical display, pointing device and keyboard.

GTK and QT are two general purpose GUI libraries that support windows, dialogs, buttons, etc.

GNOME and KDE are two desktop environments that manage the windows on the graphical desktop, provide useful applets and thingies like button bars. They also allow multiple applications to communicate through the X-server (X-terminal device) even if the apps are running on different remote computers. For instance copy and paste is a form of interapplication communication. GNOME is built on top of GTK. KDE is build on top of QT. And it is possible to run GNOME apps on a KDE desktop or KDE apps on a GNOME desktop because they all work with the same underlying X-protocol.

Answer 4

Linux on desktop and some servers is still all X and frame buffer graphics. Under X window -It comes GTK+ and Qt, YES BOTH uses X system, again there are lot of desktop environments - Gnome, KDE uses X display and their shells, etc.

Btw, There was a recent video from linux conf (http://blip.tv/file/4693305/). Keith Packard from Intel spoken about X and GL* It was interesting.