Here are just some of the things.
On systems like FreeBSD and OpenBSD, the
init program only permits logging on as user #0, in single user mode, on terminals marked as
secure in the
/etc/ttys file. And the
login program (directly on OpenBSD, through a PAM module in FreeBSD) enforces the
secure flag in multi-user mode.
Debian and Ubuntu similarly have the
/etc/securetty mechanism. In all of them, the console, and the kernel virtual terminals (which are not necessarily the console, note), default to permitting superuser log on.
On OpenBSD, the out of the box default for SSH is not to permit log on as the superuser when using password or keyboard-interactive authentication (but to permit it when using public key authetication). The out of the box default for SSH on FreeBSD is to not permit log on as the superuser at all.
framebuffer output event/USB input programs
One can tunnel X over an SSH connection, and one can interact with programs that employ a terminal device. But programs that expect to perform I/O via Linux's "event" device system, via USB HID devices, and via a framebuffer device, are not operable via SSH.
One generally doesn't get sound tunnelled over SSH, either. One can manually direct programs that make sounds to a PulseAudio server over the LAN, or manually tunnel from the remote sound clients to a local PulseAudio server. But not all sound is PulseAudio, for starters.
Programs such as BRLTTY require direct access to the character+attribute cell array of a virtual terminal device. Where a server has BRLTTY set up, it cannot access SSH login sessions and display them in Braille, whereas it can display kernel virtual terminal login sessions.
emergency mode and rescue mode work
Bootstrapping into emergency mode or rescue mode does not start an SSH server. Neither mode starts all of the hooplah that underpins networking, let alone network services like an SSH server.
unpredictable PolicyKit stuff
PolicyKit has the notion of active and inactive login sessions. This tries to boil down what in the
/etc/ttys system is a set of several attributes (
dialup) into a single true/false switch between "active" and "inactive". This doesn't quite fit the world where SSH and its ilk exist.
One can only have an "active" login session when one logs on on the console or on a kernel virtual terminal. Logging on via SSH is always considered to be an "inactive" login session. Surprising behavioural differences can result, because software authors and the system administrator can grant differing permissions to do stuff to active versus inactive login sessions.