Yes, it is impossible to recover the private key from the public key. If it was possible, RSA would be fundamentally broken, and this would be major news (breaking RSA would not only break a lot of Internet communication security, but also allow all kinds of banking fraud, amongst others).
Logging in with a public key instead of a password in fact tends to increase security. If your password isn't strong enough, it can be brute-forced by an attacker with sufficient bandwidth. If the attacker doesn't have a copy of your private key file, the RSA key effectively cannot be brute-forced (a 1024-bit key is equivalent to something like a 160-character password made of random case-sensitive letters and digits). Someone who watches over your shoulder might be able to see your password and your key passphrase, but with a key they'd also need to get the key.
Private keys aren't always more secure than passwords. If the attacker obtains a copy of your private key files (for example by stealing your laptop or your backup media), she can try to brute-force the passphrase, and she can do it at high speed since you have no way to limit the rate (unlike password guesses that need to be made online). If your passphrase is good enough and you notice the theft immediately, you'll still have time to revoke the key.
A public key introduces an element of privacy exposure: if someone knows that you've used the same public key to log into A and to log into B, they know the same person logged into A and B. Merely possessing the public key makes you a suspect that you also have the private key, so you lose some anonimity. But that's usually minor, especially if you're just storing the key in ~/.ssh where only system administrators (who also know what IP address you logged in from) can see it.
These security considerations aside, a private key has many practical advantages. You don't need to type your password so often, and in particular can run automated scripts that don't prompt you once you've entered your key in ssh-agent or the like. You don't need to type your password so often, so you can afford to make it higher-entropy (longer, harder to type). You don't need to type your password so often, so there's less risk that it'll be snooped by a human observer or camera.
