This isn't an answer, so if you find one, obviously that is preferable.
If not, and this is the root cause of your conundrum:
The reason I am not ssh'ing directly is because I don't want that user to be accessible through ssh (it is the "virtualbox" user, which is obviously an easy target for bots trying to ssh to that server)...
It seems to me not a great piece of reasoning. "What bots target" WRT a well configured sshd is largely irrelevant. Will they buzz around port 22 all over the place anywhere? Apparently so. Does this mean they actually have a chance of success?
Try googling around for a story about someone who's had a random anonymous bot successfully break into sshd. I'm sure it must have happened to someone somewhere (and, of course, you might never know), but I can't find any reports of such. It would be interesting to read what the configuration used was.
SSH is very secure when used properly. If it weren't, internet commerce would not be feasible. So why do these bots bother? By "used properly" I mean, primarily, with mandatory public/private key pairs. If you do that and are confident your private key is secure (and you should be), be confident about sshd.
I think the reason all the "break-in attempts" happen at all is that there is a large body of users who don't do things like ask questions on U&L ;) and don't read manual pages and use password protection alone, which is sort of like leaving your ATM card in a machine somewhere with a sign saying, "Guess away!".
However, if you think of your private key like your ATM card -- as something that is physically secured by you (which it essentially is), then the target becomes much much more ethereal. All those bots can do is prove that yes, it really would take thousands of machines thousands of years working together to brute force a 2048-bit key.
If you are sick of reading reports of break-in attempts, change your port. I have seen people here poo-poo this as "security by obscurity", however, an sshd which is properly secured on port 22 is not going to be any less secure on port 57, but it won't be randomly bothered. Of course, all your drone enemies could just port scan the whole IP -- but you know what? They don't. I'd presume this is because what they are looking for is someone running a system who hasn't even looked at
/etc/ssh/sshd_config, much less educated themselves and tuned it.