If I would install xserver and a window manager without installing an display driver (xorg-video-intel for example) would the performance suffer a lot? Would I actually see or feel a difference in:
- browsing the internet
- watching video files
An X server without a display driver would not be able to display anything. This exists, for example Xvfb, which “displays” to a memory buffer. This has its uses, such as to perform automated processing (e.g. testing) involving programs with a graphical user interface, but you won't see anything with it, by definition.
If you meant that you don't want to install a display driver that's specific to your graphics card, and you're using a PC, then you can use the Xorg VESA driver. This is a display driver, it just happens to be one that works on pretty much any graphics card found in a PC. It's quite limited, I think you can't go above 800x600 resolution, and performance is visibly bad even for basic things (forget about videos or 3D).