I'd like to block physical console access to my linux. I'd like to find, which module is responsible for keyboard input (ps/2, usb, ...) and vga output (only physical), so I can remove them from my kernel (recompile) and then only have access through serial or through ssh.
Once someone has physical access to the system, there is little you can do. Your best bet would be something like TrueCrypt on all your hard drives, and as we'll as removing the CD and floppy drives as stated in an above answer, hot glue over the SATA/IDE connectors to prevent anyone plugging a CD drive in . Set a really good password on BIOS/ UEFI and turn on tamper detection if your computer supports it. Hot glue the bios reset jumper pins and hot glue or desolder the USB and PS/2 connectors. Glue in the RAM and/or power cable to prevent cold boot attacks, then hot glue, solder or weld the case shut.
After this, software won't matter.
How completely do you want to block console access? If it is just for when the system is up and running then you can simply disable getty from running. Look at /etc/init/console.conf and /etc/init/tty?.conf. If you want to make sure that nobody reboots into a live cd then your best bet is to epoxy the USB ports and remove the cd drive from the system.