Gentoo Linux offers a tarball called a stage3 archive with only a base layout and tool chain which is basically what they use to build their livecd (minimal CD not livedvd). Since Gentoo Linux is a meta distribution and it's source base you will be able to install any GUI administrative utilities that you need, it also supports every major DE/WM out there but if you want just a headless systems then the stage3 tarball will be enough and all you'll have to install to get a live system going is your bootloader and kernel.
Linux From Scratch is a bit like Gentoo in terms of software that is compiled from source but the main difference is that LFS doesn't provide a package manager and forces you to manage all dependencies on your own (great learning experience BTW), so if you just want a system that provides both the flexibility to tweak what features a package will be installed with through use of a package manager then go with Gentoo, otherwise use LFS or Arch Linux.
Arch Linux is between Gentoo and a binary distribution like Ubuntu but with less bloat.
Seems like you want a distro to that you control. I say either Gentoo or Arch Linux. GL