xserver-xorg pulls in
xserver-xorg-video-all, which pulls in all the video drivers.
If you know which driver is appropriate, you can install it along with
xserver-xorg, and it will satisfy the alternative dependency on
xorg-driver-video, avoiding the
xserver-xorg-video-all dependency altogether; thus for an Intel IGP:
apt-get install xserver-xorg xserver-xorg-video-intel
If you want to make doubly sure that
xserver-xorg-video-all isn't installed, you can use something like
apt-get install xserver-xorg xserver-xorg-video-intel xserver-xorg-video-all_
(note the underscore, which instructs
apt-get to purge the package).
To avoid installing the
libdrm* packages, you need to avoid
libgl1-mesa-dri which depends on all of them (with no easy workaround, so if you want DRI support you need all three implementations, unless you use
equivs — but they're quite small).
xserver-xorg recommends the package, so
apt-get --no-install-recommends install xserver-xorg ...
should do the trick. This is a good idea anyway if you're after a small image.