Advice for managing SSH keys

Question

What is the best practice you have found for managing lots of SSH keypairs?

I use SSH to connect to several systems, both at home and at work. I currently have a fairly small, manageable collection of keypairs for both work and home systems. I have a script that generates a named keypair so I can avoid confusion.

My home network is comprised of my laptop (ubuntu), two desktops (ubuntu/fedora dual boot, fedora/windows dual boot), and a media system (ubuntu). At work I have my personal laptop (which I use for working from home), my desktop (fedora), a production system (RHEL), and an a laptop with windows (sigh) and a VM (fedora). All good so far.

(I have no interest in either putting my home keypair on my work system, or my work keypair on my home systems. And we have virtual user accounts to mechanize file transfers with other systems, where the private key must reside on the production machine, to transfer files between other systems.)

But now comes Hadoop, a large cluster of 100+ systems, and with that more complexity, more users, and more keypairs. Now I need to manage keys.

(I need to clarify. I am a software developer consulting to a client who is deploying a Hadoop cluster. They need to manage keys. There will be many folks accessing the cluster, needing to place their public keys onto the system. As the resident Linux savant, they asked me for help. I advised hiring a system admin, but until they do, I am helping)

When I need to publish the public key to a remote system, all of the 'how-to' web pages suggest either overwrite (>) (destroying existing keys), or append (>>) (which is good, preserves existing keys). But I would think preserving each public key on the destination machine separately, and combining them would be better. I am looking for advice.

What is the best practice you have found for managing lots of keys?

Thank you!

---

Edit: One aspect is needing to place keys on lots of systems, and the concomitant CRUD (create, read, update, delete/disable) for specific users, which means needing to be able to identify which keys belong to which users.

Answer 1

Generally you should not have more than 1 key per client machine (emphasis on "generally"). I'm not sure if I'm understanding your question properly, but if you're saying you have a separate key for every remote system, then you're definitely doing it wrong.

Ssh uses public key cryptography. The key you install on the remote system is the public key, there is absolutely no harm in reusing this key elsewhere. It's the private key that must be protected and remains on your personal system.

It's also a good idea to have the private key reside on only one client, not shared. This is so that if a client is compromised, you can revoke just that one key.

---

Now, if you're asking how you can get your public key out to hundreds of systems there are a couple ways of doing this.

The most common way is by using shared home directories. Have an NFS (or other network filesystem) mounted (or automounted) all all the systems.

The other way is to take advantage of a new feature in ssh. It's a configuration directive called AuthorizedKeysCommand. Basically it's a command that sshd will run every time it needs to look up a public key. The command simply writes the public key of the user being queried to it's STDOUT. This is primarily used when you don't have mounted home directories, but still have a central authentication server (FreeIPA takes advantage of this).

Of course you can do other things such as cron job rsync on /home from a central server. But that's not a common practice.

Answer 2

When I need to publish the public key to a remote system, all of the 'how-to' web pages suggest either overwrite (>) (destroying existing keys), or append (>>) (which is good, preserves existing keys). But I would think preserving each public key on the destination machine separately, and combining them would be better. I am looking for advice.a

i do not see any advantage in storing the public keys both in .ssh/authorized_keys and in a separate file.

if you have a look at the actual keys stored in the *authorized_keys* file you see that they already contain human-readable meta-information about the origin of the key. e.g. the public key for user@foo usually has an entry like:

ssh-rsa AAAAB3Nza...LiPk== user@example.net

thus making it very easy to inspect/extract/delete certain keys (attached to certain users) from the *authorized_keys* file.

the user id is really a free-form "comment" field, so you can put any information in there, that you think necessary to identify the given key.

in any case, you should only generate key-pairs for "users" that need to access remote ressources. chances are, that you don't need to login from each hadoop host to each other hadoop host. rather you will have a few management machines, that need to access all the hadoop hosts. you only need a single key-pair per management machine, and install each public-key on all hadoop hosts.

Answer 3

Recent OpenSSH allow to get ssh keys from LDAP, see 'AuthorizedKeysCommand' in sshd_config man page. Personally I would prefer OpenSSH certificates, not keys, see http://blog.habets.pp.se/2011/07/OpenSSH-certificates. You can manage the keys with any configuration manager like cfengine, puppet, chef, salt...

Answer 4

An other way that is super easy to implement and allows a bit more flexibility in terms of adding multiple users is to use. https://userify.com/

It allows you to define different groups of servers and have keys for different users enabled or disabled for those servers.

Super simple installation and management.