I read in quite a few places that using a netinst CD was the best way to get a minimal installation of Debian. I chose only the LXDE option on the installation page, not even installing 'standard system utilities' but I still got Libreoffice and GIMP.
If the smallest ISO I could find installed these by default, how do I install the OS without?
You chose the LXDE desktop environment option. This installs task-lxde-desktop, which pulls in, among other things, packages lightdm, lxde, firefox, libreoffice, lxlauncher, synaptic, and xorg, including a browser and office suite as part of a typical desktop load.
If you want a minimal installation, don't select additional packages to install at this point. As it says on that page of the installer:
At the moment, only the core of the system is installed. To tune the system to your needs, you can choose to install one or more of the following predefined collections of software.
That is, you have a working core system at that point, and it's offering you the opportunity to install some more things that it thinks are coherent sets of packages for particular uses. Otherwise, you can carry on from there with the minimal install you already have.
If you leave the menu blank:
then no further packages will be installed. You have to unselect "Debian desktop environment" explicitly. You can later install whichever packages you need on top of that specifically after booting into the system.
I'm assuming you want a minimal LXDE desktop environment. To do that, deselect everything except "Standard system utilities" in tasksel, you can deselect that too if you want, though some of them are useful and they take little space.
Then after booting your new command-line only system (as root, su or sudo depending how you're configured) issue:
apt-get install lxde-core
You should be good to go.
The type of install image does not matter for debian, once you connect them to the web you can install everything. As far as that goes if you have ipxe already you can install debian with nothing more than a 64k rom image and a network image and a network cable. you can also use a full set of CDs (or DVDs or Blu Rays) to install a minimal system. How minimal? Unless you do preseeding the packages with a priority higher than standard and their requirements and recommendations is as small as you go if you select nothing in the task selection. If you want smaller than that you need to use a preseed file which you can use to override package selections, blacklist individual packages and ignore recommendations.
Now if you want to shrink your installation you can either do it again (I have done so probably about five times over the past couple days, but I'm developing a udeb) or just fire up aptitude and remove what you don't want (I recommend aptitude, it is amazing).
