SSH certificate authentication (PKI) uses a private key, which is private, and a public key, which does not need to be kept secret. When you set up a connection for PKI, your private key remains securely with you on the client and your corresponding public key is placed on the server. You can have multiple public/private key pairs and tell ssh which one to use (either on the command line with ssh -i {identity_file} or in the configuration file per host block).
AWS probably generates the public/private key for a login to the new instance as a convenience. You download the private key over a secure connection and store it locally. It already has the public key so from this point forward the combination of your copy of the private key and its copy of the public key allow verification of the connection attempt.
When you run the command ssh -i {private_key} {remote_host}… what's actually happening is that the ssh command uses this private key as part of the negotiation to establish the connection. It's unlikely anything else will have the public key unless you have copied it to a different target.
You can place the same public key on as many remote machines as you like. Your instance of the private key can then verify your access to any of these remote machines.
You should not share the private key by copying it to other servers. Instead, where feasible you should generate a separate public/private key combination for each place that you want to treat as a client (i.e. from where you initiate an ssh connection), and copy the newly created public key to the appropriate target servers.
Finally, you ask, "Can I connect to remote server in other clouds or virtual machines located on my PC like I do it in AWS - only with private key of remote machine?" You can only connect to a remote server that has a public key corresponding to one of your private keys.
