3

I created a new AWS EC2 instance, using the AMI image, configured it to my liking, and saved an image, (Call this svr01). Then I created another EC2 instance, using that same image ((Call this svr02). Both instances have elastic IPs, which I have an entry for in an external hosted zone, DNS Lookup and ICMP traffic works fine. I also created DNS records for the servers on an internal private zone, using the private IPs, and again, the DNS/ICMP works fine from inside the private zone.

So here's the weird part. I restarted both svr01 and svr02, and both came up, but now I can only ssh in externally to svr01... svr02 just hangs. Here's the the ssh connection attempt, very very verbose output:

me$ ssh -vvv svr02.mydomain.tld
OpenSSH_6.2p2, OSSLShim 0.9.8r 8 Dec 2011
debug1: Reading configuration data /Users/me/.ssh/config
debug1: Reading configuration data /etc/ssh_config
debug1: /etc/ssh_config line 20: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to svr02.mydomain.tld [123.123.123.123] port 22.
debug1: Connection established.
debug3: Incorrect RSA1 identifier
debug3: Could not load "/Users/me/.ssh/id_rsa" as a RSA1 public key
debug1: identity file /Users/me/.ssh/id_rsa type 1
debug1: identity file /Users/me/.ssh/id_rsa-cert type -1
debug1: identity file /Users/me/.ssh/id_dsa type -1
debug1: identity file /Users/me/.ssh/id_dsa-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.2
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1
debug1: match: OpenSSH_6.6.1 pat OpenSSH*
debug2: fd 3 setting O_NONBLOCK
debug3: load_hostkeys: loading entries for host "svr02.mydomain.tld" from file "/Users/me/.ssh/known_hosts"
debug3: load_hostkeys: found key type RSA in file /Users/me/.ssh/known_hosts:21
debug3: load_hostkeys: loaded 1 keys
debug3: order_hostkeyalgs: prefer hostkeyalgs: ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa
debug1: SSH2_MSG_KEXINIT sent

Now I can login to svr01 just fine, and from svr01, I can login to svr02 (using the same account I'm trying to use when logging in externally). So from there, I looked at the logs when I try to ssh in externally, which I find:

Feb 23 21:40:49 svr02 sshd[2608]: debug1: Forked child 2647.
Feb 23 21:40:49 svr02 sshd[2647]: Set /proc/self/oom_score_adj to 0
Feb 23 21:40:49 svr02 sshd[2647]: debug1: rexec start in 5 out 5 newsock 5 pipe 7 sock 8
Feb 23 21:40:49 svr02 sshd[2647]: debug1: inetd sockets after dupping: 3, 3
Feb 23 21:40:49 svr02 sshd[2647]: Connection from 75.172.150.140 port 57827 on 172.31.29.9 port 22
Feb 23 21:40:49 svr02 sshd[2647]: debug1: Client protocol version 2.0; client software version OpenSSH_6.2
Feb 23 21:40:49 svr02 sshd[2647]: debug1: match: OpenSSH_6.2 pat OpenSSH* compat 0x04000000
Feb 23 21:40:49 svr02 sshd[2647]: debug1: Enabling compatibility mode for protocol 2.0
Feb 23 21:40:49 svr02 sshd[2647]: debug1: Local version string SSH-2.0-OpenSSH_6.6.1
Feb 23 21:40:49 svr02 sshd[2647]: debug1: SELinux support disabled [preauth]
Feb 23 21:40:49 svr02 sshd[2647]: debug1: permanently_set_uid: 74/74 [preauth]
Feb 23 21:40:49 svr02 sshd[2647]: debug1: list_hostkey_types: ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
Feb 23 21:40:49 svr02 sshd[2647]: debug1: SSH2_MSG_KEXINIT sent [preauth]
Feb 23 21:40:49 svr02 sshd[2647]: debug1: SSH2_MSG_KEXINIT received [preauth]
Feb 23 21:40:49 svr02 sshd[2647]: debug1: kex: client->server aes128-ctr hmac-md5-etm@openssh.com none [preauth]
Feb 23 21:40:49 svr02 sshd[2647]: debug1: kex: server->client aes128-ctr hmac-md5-etm@openssh.com none [preauth]
Feb 23 21:40:49 svr02 sshd[2647]: debug1: kex: diffie-hellman-group-exchange-sha256 need=16 dh_need=16 [preauth]
Feb 23 21:40:49 svr02 sshd[2647]: debug1: kex: diffie-hellman-group-exchange-sha256 need=16 dh_need=16 [preauth]
Feb 23 21:40:51 svr02 sshd[2647]: Connection closed by 75.172.150.140 [preauth]
Feb 23 21:40:51 svr02 sshd[2647]: debug1: do_cleanup [preauth]
Feb 23 21:40:51 svr02 sshd[2647]: debug1: do_cleanup
Feb 23 21:40:51 svr02 sshd[2647]: debug1: Killing privsep child 2648

I thought maybe it would be something silly like the /etc/ssh/ssh_host_rsa_key being the same key since it was a cloned image, but they are different.

And just to rule out the security groups, I temporarily added it to a group that allows all traffic, in and out, no restrictions at all.

Anyone have any ideas?

Update

So after I created this post, I stepped away for several hours, and logged on now, and it's resolved... If anyone knows what could cause this, let me know, in case it happens again.

closed as off-topic by Jakuje, Stephen Kitt, Anthon, garethTheRed, muru Feb 24 '16 at 11:20

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Questions describing a problem that can't be reproduced and seemingly went away on its own (or went away when a typo was fixed) are off-topic as they are unlikely to help future readers." – Jakuje, Stephen Kitt, Anthon, garethTheRed, muru
If this question can be reworded to fit the rules in the help center, please edit the question.

  • What is the difference between the distros? Compare openssh versions, sshd_config. You might also set up verbose log on server LogLevel DEBUG3. But it looks like somewhere on the way the long packets containing the Kex information gets lost. Ref – Jakuje Feb 23 '16 at 22:20
  • I stated that I setup svr01, created an AMI image from it, then created another EC2 instance from that image, so the versions are identical. And this just randomly resolved itself... I updated the post – Justin Feb 24 '16 at 3:43
  • I have had all sorts of problems with amazon internal traffic. mysql servers not allowing connection on 172 private on the mysql port being the main, but right now just trying to rsync a media folder between servers. Because support is expensive or non existent you are left guessing. – MagicLAMP Sep 4 '18 at 13:20

Browse other questions tagged or ask your own question.