Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I work on remote servers frequently, but for some reason I cannot discern, today I am unable to ssh into anything.

Everything comes back with Write failed: broken pipe.

I know this is a common symptom of server timeouts, but I can't do anything besides enter a password.

When connecting, the password dialog is normal, and does recognize invalid attempts. However, as soon as the correct password is entered, the broken pipe hits.

I did try apt-get install --reinstall openssh , but it didn't fix anything.

Any thoughts?

This is on ubuntu 11.04, but ditching Gnome for OpenBox.

--

ssh -v output:

~ >> ssh -v clearpoint@turnleftllc.com
OpenSSH_5.8p1 Debian-1ubuntu3, OpenSSL 0.9.8o 01 Jun 2010
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to turnleftllc.com [72.167.39.231] port 22.
debug1: Connection established.
debug1: identity file /home/tom/.ssh/id_rsa type 1
debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
debug1: identity file /home/tom/.ssh/id_rsa-cert type -1
debug1: identity file /home/tom/.ssh/id_dsa type 2
debug1: Checking blacklist file /usr/share/ssh/blacklist.DSA-1024
debug1: Checking blacklist file /etc/ssh/blacklist.DSA-1024
debug1: identity file /home/tom/.ssh/id_dsa-cert type -1
debug1: identity file /home/tom/.ssh/id_ecdsa type -1
debug1: identity file /home/tom/.ssh/id_ecdsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.5
debug1: match: OpenSSH_5.5 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.8p1 Debian-1ubuntu3
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: RSA 87:81:14:42:23:b7:5b:94:eb:a9:f5:25:e0:e9:1a:0b
debug1: Host 'turnleftllc.com' is known and matches the RSA host key.
debug1: Found key in /home/tom/.ssh/known_hosts:9
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure.  Minor code may provide more information
Credentials cache file '/tmp/krb5cc_1000' not found

debug1: Unspecified GSS failure.  Minor code may provide more information
Credentials cache file '/tmp/krb5cc_1000' not found

debug1: Unspecified GSS failure.  Minor code may provide more information


debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/tom/.ssh/id_rsa
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug1: read PEM private key done: type RSA
debug1: Authentication succeeded (publickey).
Authenticated to turnleftllc.com ([72.167.39.231]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
Write failed: Broken pipe

Trying the above with a different server yielded almost identical results.

For good measure, here's -vvv using a different server.

debug2: we did not send a packet, disable method
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/tom/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,gssapi-with-mic,password
debug1: Offering DSA public key: /home/tom/.ssh/id_dsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,gssapi-with-mic,password
debug1: Trying private key: /home/tom/.ssh/id_ecdsa
debug3: no such identity: /home/tom/.ssh/id_ecdsa
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
koenig@50.57.55.206's password: 
debug3: packet_send2: adding 48 (len 63 padlen 17 extra_pad 64)
debug2: we sent a password packet, wait for reply
debug1: Authentication succeeded (password).
Authenticated to 50.57.55.206 ([50.57.55.206]:22).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug1: Entering interactive session.
debug2: callback start
debug2: client_session2_setup: id 0
debug2: fd 3 setting TCP_NODELAY
debug3: packet_set_tos: set IP_TOS 0x10
debug2: channel 0: request pty-req confirm 1
debug1: Sending environment.
debug3: Ignored env ORBIT_SOCKETDIR
debug3: Ignored env SSH_AGENT_PID
debug3: Ignored env TERM
debug3: Ignored env SHELL
debug3: Ignored env XDG_SESSION_COOKIE
debug3: Ignored env WINDOWID
debug3: Ignored env GNOME_KEYRING_CONTROL
debug3: Ignored env GTK_MODULES
debug3: Ignored env USER
debug3: Ignored env LS_COLORS
debug3: Ignored env LIBGL_DRIVERS_PATH
debug3: Ignored env SSH_AUTH_SOCK
debug3: Ignored env USERNAME
debug3: Ignored env DEFAULTS_PATH
debug3: Ignored env XDG_CONFIG_DIRS
debug3: Ignored env PATH
debug3: Ignored env DESKTOP_SESSION
debug3: Ignored env PWD
debug3: Ignored env GDM_KEYBOARD_LAYOUT
debug3: Ignored env GNOME_KEYRING_PID
debug1: Sending env LANG = en_US.UTF-8
debug2: channel 0: request env confirm 0
debug3: Ignored env GDM_LANG
debug3: Ignored env MANDATORY_PATH
debug3: Ignored env UBUNTU_MENUPROXY
debug3: Ignored env PS1
debug3: Ignored env GDMSESSION
debug3: Ignored env SHLVL
debug3: Ignored env HOME
debug3: Ignored env LANGUAGE
debug3: Ignored env LOGNAME
debug3: Ignored env XDG_DATA_DIRS
debug3: Ignored env DBUS_SESSION_BUS_ADDRESS
debug3: Ignored env LESSOPEN
debug3: Ignored env WINDOWPATH
debug3: Ignored env DISPLAY
debug3: Ignored env LESSCLOSE
debug3: Ignored env COLORTERM
debug3: Ignored env XAUTHORITY
debug3: Ignored env _
debug2: channel 0: request shell confirm 1
debug2: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
Write failed: Broken pipe
share|improve this question
    
Is this connecting to one server or muliple servers? What is the state of your internet connection? Having you checked to determine that there isn't substantial congestion or packet loss using traceroute and ping? –  gview Apr 17 '12 at 1:30
    
It is to any server. My internet connection is fine; I have tried both wireless and wired, and have pinged my primary dev server with mtr and had no anomalies or packet loss. –  Tom Thorogood Apr 17 '12 at 1:45
1  
Add one or more -v flags to the ssh command, and include the results in your question. –  Warren Young Apr 17 '12 at 2:45
1  
@TomThorogood: Wow, that is seriously weird. Your logs show the SSH client negotiating the connection successfully, then it gets killed for no reason the client can diagnose even at -vvv. That means the bad actor is an outside agent. A broken firewall, maybe. Can you ssh between two boxes plugged into the same network switch? –  Warren Young Apr 17 '12 at 15:56
1  
Sorry i haven't responded at all today. It does indeed seem to be a network issue. It's fine elsewhere, but when I'm at home, nothing. Sadly, our building provides the internet, and they're a pretty crappy provider -- the speed is great, but the service not so much. So I'll have to wrestle with them to get them to fix it. Though I'm not sure what I should ask them for. –  Tom Thorogood Apr 17 '12 at 23:36

3 Answers 3

up vote 1 down vote accepted

When I have seen this kind of behavior, it's usually something network related.

You didn't mention if you were connecting from work or home. If you are connecting from home, I would reboot the personal router connecting to the internet. If you are connecting from work, I would check with the networking group and find out if something has changed. A new IPS/IDS? New definitions or thresholds for the IPS/IDS? New firewall policy? New timeout value?

I feel like you may want to start looking outside of SSH.

share|improve this answer

Use editor for editing /etc/ssh/ssh_config and comment out the GSSAPI* options.

EDIT1:

I've found, that the same behavior is, when you are connected via some D-link modems, you can find it here, here or here. You can solve this adding this line to firewall rules:

iptables --table mangle --append OUTPUT --jump DSCP --set-dscp 0x0

If your server act as router, you will need also this line:

iptables --table mangle --append PREROUTING --jump DSCP --set-dscp 0x0
share|improve this answer
    
No. In the -v logs above, you do see it complain about GSSAPI, but that's just because it tries that first. It fails, and successfully pushes past it to other options. There's nothing wrong with this. If you have a Kerberos setup, you do want it to be used in preference to ad hoc PSK. –  Warren Young Apr 17 '12 at 15:54
    
OK, you are right. Can you check out of logs on the remote server and settings in the /etc/ssh/sshd_config? –  Jan Marek Apr 18 '12 at 6:09
    
Please, try a new found solution... –  Jan Marek Apr 19 '12 at 10:32

check auth.log, syslog, or messages. Canned answer, but it's worth a shot.

share|improve this answer
    
The only thing that revealed anything at all was syslog, which shows "Session opened by Tom", followed immediately by "Session closed by Tom". –  Tom Thorogood Apr 17 '12 at 9:02

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.