I'm currently using postfix and SASL on my personal server for authenticated SMTP.
The server is purely for my personal use and my personal domains, so I'd much rather have something simpler based on ssh public keys. Does any such solution exist?
While it isn't SSH's pubkey authentication (which is something that only exists in the SSH protocol, not SMTP), you could set up TLS Client certificates. This will require a valid SSL certificate on the client side.
Also, if you must use SSH's pubkeys, you could simply allow all mail connections from localhost on your personal SMTP server, and set up an SSH tunnel over SSH to port 25 on the SMTP server.
Nothing prevents networked application to use ssh keys for authentication/encryption, except these applications have to be written to support this (i.e. by using libssh). You don't say what mail user agent you use, nor what SMTP server it connects to, but it is not very likely they support ssh natively.
But of course, you can use a normal ssh connection to your server to make a tunnel for SMTP sessions.This of course authenticates the users to your machine, not to your SMTP server, which may not be what you want.
SMTP servers usually don't check the SSL certificates when connecting, but do use them to encrypt the channel. This makes them easy to configure with self-signed certificates, or certificates from a private authority. (Self-signed certificates are equivalent to ssh certificates.) I use tinyCA to create my own certificate authority. You can use key sizes up to 4096. You should use a size of at least 2048.
MUAs (Mail User Agents: Firefox, Outlook, etc) work well with self signed certificates. You do need to accect them the first time. I suggest using the submission port with startTLS and authentication. This will give you a secure authenticated channel.
I use Dovecot IMAP with startTLS using self-signed certificates for reading email. Using IMAP gives extra options like using a WebMail interface in addition to the one or more clients.
Configure your mail server to allow unauthenticated relay from localhost and set up an ssh tunnel for sending mail.
ssh yourserver -L 8587:localhost:587
This will forward the local port 8587 through ssh to yourserver on yourserver's port 587. Then configure your mail client to use
Although I would still encourage you to leave SASL authentication on.