a private key needs to be stored on every computer you use for SSH.
The second paragraph (specifically, I'm focusing on this part, that I quoted above) is correct: if you log in from multiple machines, you'll need to have the private key accessible from multiple machines. This means a few things. First, each machine that you log in from will need to be trusted to not do terrible things with your private key. Malicious machines cannot be trusted. Second, you'll need to figure out a way to get your private key onto those machines. That may be inconvenient.
There are a couple of ways around this, if you want to be able to log in from multiple machines. One way is to log in from a less secure machine to a secure machine, and have the secure machine store the key(s) that work with various other systems. This still leaves a weak link in the security chain, because then an attacker could try to get your password to the middle machine. Once that is obtained, the attacker could probably access the keys that get you into other systems. However, the good news is that you're not needing to place your private key on every computer you go to, so this does offer a convenience factor.
Another way is to use one-time credentials. (More commonly known as: one-time passwords.) With this method, you can use a trusted device to generate a passphrase. I know some people have used their smart phones. Then, you really don't care if your key is compromised (stolen/copied) after you have successfully used the passphrase, because the passphrase isn't any good after that anyway.
Working with one-time passphrases may be slightly more convenient if you typically log into the same system, as some software might remember some details (like a sequence number) which would require less manual work on your part. You could combine these methods: use a one-time passphrase at the start of your SSH activity, and log into a trusted system. Then reference the key files on that system to log into other systems.
Therefore if you want the ability to SSH into a server from any computer you should stick with username and password.
Therefore, the third paragraph is a part of your premise that I'm not agreeing with, because I interpret your question as meaning that you should stick with the way you've been doing things, using a standard password. Instead, use the passphrases that are part of the "one-time password" methods. This way, you use a more complex security method (providing better security, like what key files do), but use one that doesn't require untrusted machines to use key files.
To point you in a right direction, OpenSSH supports one-time passwords via the skey method, using a program called skeyinit. Then, to generate the passphrases on a mobile device, OTPdroid for Android also supports this. There may be other workable solutions, but this is simply describing some resources that are known to work well with each other.
On a side note: with the situation you're describing (multiple systems), I highly recommend that you get familiar with terminal multiplexers (tmux or screen). It will let you do some work (for instance, logging in from the middle system into other systems), and then detach a session. Then, after you perform the initial login from another machine, you can resume a detached session, and some of your mundane work (like logging into other machines) might not need to be done again.