A simple and in practice effective protection against scan-based attacks is not to use the standard port. 443 (the https port) exposes you to different brute-force attacks that aren't going to crack your weak passwords, and possibly works through more firewalls than the default port (22).
Most methods to prevent ssh brute force attacks are great ways to self-DoS (oops, I screwed up the configuration! Oops, I did a bunch of quick rsync's and am now banned for the day!) or assisted-self-DoS (Oops, the attacker comes from/has subverted a machine in the same subnet as me (dynamic IP range, college network...) and I'm getting banned as well!).
If you only log in from a few places, you can just whitelist source IP addresses. That's obviously no good if you want to ssh from your laptop or cell phone on the go.
Having an ssh daemon that only listens to IPv6 connections should protect you from scans for a few years yet. But many firewalls won't let you transport IPv6 in any reasonable way.
Another method you don't mention is port knocking. It doesn't suffer from as self-DoS problems (other than misconfiguration), but it doesn't cross firewalls well, and can add several seconds' latency to connection establishment.
If you have good passwords, or you can live without password authentication, disable password authentication. (Keys and one-time passwords are sufficient for most use cases: if you don't trust the client machine enough to store an ssh key, you don't trust it not to have a keylogger either). Then brute force attacks will cost you a bit of CPU and bandwidth but don't expose you to an intrusion (as long as you've checked none of your keys came from a Debian low-entropy OpenSSL).
All in all, do note that changing the port does not significantly reduce your exposure. You'll get less scanning, but all you can cut off is the low-hanging fruit that seeks to exploit old vulnerabilities and weak passwords. As long as you keep your daemon up to date and either enforce reasonable passwords or and reasonable attempt rate limits, switching the port is more of a liability than a security measure.