19

The most recent thing I remember is changing the soft and hard memlock ulimit to unlimited. Now I can't ssh into the machine.

This is the ssh log.

Authenticated to IP ([IP]:22).
debug1: channel 0: new [client-session]
debug2: channel 0: send open
debug1: Requesting [email protected]
debug1: Entering interactive session.
debug2: callback start
debug2: fd 3 setting TCP_NODELAY
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug1: Sending environment.
debug1: Sending env LC_CTYPE = 
debug2: channel 0: request env confirm 0
debug2: channel 0: request shell confirm 1
debug2: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel 0: rcvd adjust 2097152
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0
Last login: Wed Aug  6 07:18:07 2014 from IP-SOURCE
debug2: channel 0: rcvd eof
debug2: channel 0: output open -> drain
debug2: channel 0: obuf empty
debug2: channel 0: close_write
debug2: channel 0: output drain -> closed
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype [email protected] reply 0
debug2: channel 0: rcvd eow
debug2: channel 0: close_read
debug2: channel 0: input open -> closed
debug2: channel 0: rcvd close
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send close
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
Connection to IP closed.
Transferred: sent 4256, received 2504 bytes, in 0.4 seconds
Bytes per second: sent 9616.9, received 5658.0
debug1: Exit status 254

I have tried the following unsuccessfully till now, before posting here:

  1. Trying for a norc noprofile login by ssh user@host 'bash --noprofile'

  2. Forcing a tty by ssh -t user@host

  3. Moved the bash_profile. Tried sshing by ssh user@host.

  4. Renaming the limits.conf file in hopes that it won't be read.

  5. Restarted ssh server.

  6. Run a command via knife as knife ssh "name:server" "come_command"

  7. ssh user@host 'ulimit -l 64', ssh user@host 'ulimit -S -l 64', ssh user@host 'ulimit -H -l 64', ssh user@host 'exec ulimit -H -l 64'

I am not sure if this way of running commands inline: ssh user@host "some_command" works, because I can't get a simple directory listing. I also tried rebooting by ssh user@host 'reboot' but don't think the command was executed. I restarted the machine from AWS also, but unsuccessful.

Is it a lost cause trying to ssh? Is there any way I can ssh into the server?

8
  • 1
    If the login shell of the remote user is bash, then bash always reads ~/.bashrc over ssh. There's no way around that. Can you sftp there and check/alter your bashrc that way? Commented Aug 6, 2014 at 12:07
  • Tried sftp using CyberDuck, got the SSH_FXP_INIT error code. Commented Aug 6, 2014 at 12:09
  • You can also get more detailed output using the -v option, or for more, the -vv option of for even more, then -vvv option. E.g. ssh -vvv user@host. That may give you a better idea of where things are going wrong.
    – Warwick
    Commented Aug 6, 2014 at 12:43
  • Tried that. This is where I got the log from. Commented Aug 6, 2014 at 13:13
  • Do you have any other file (FTP/HTTP...?) or shell (console?) access to the machine which you can use? Commented Aug 6, 2014 at 15:14

6 Answers 6

17

Try to change

UsePAM yes

on

UsePAM no

in /etc/ssh/sshd_config (for CentOS)

7
  • 2
    It works, but why?
    – FelikZ
    Commented Jul 7, 2015 at 11:12
  • 1
    sorry, but i dont remember the reason) Commented Jul 8, 2015 at 11:53
  • Is SELinux in play here ? Wondering if the context of /etc/security/limits.conf got hosed, and pam can't use it any more.
    – steve
    Commented Jul 15, 2016 at 19:46
  • If you are using a distribution using systemd this is a bad solution IMHO. This will prevent logindto a open a session and when/if the user is rebooting the machine, some processes started as the user will not be stopped as expected.
    – Bigon
    Commented Dec 19, 2016 at 19:30
  • 1
    In my case, the reason is the user Hard Open Fils Limit per processor is bigger than the nr_open. (nr_open is reset when restarting the machine): you can check the nr_open by cat /proc/sys/fs/nr_open, and hard open files ulimit -Hn. If you still want the ssh login user use the Hard Open File Config, you need to increase nr_open: sudo sysctl -w fs.nr_open=NUM_BIGGER_THAN_HARD
    – Xin Meng
    Commented Mar 22, 2019 at 14:14
6

I had a similar problem, I seemed to only see following strange message:

client_input_channel_req: channel 0 rtype exit-status reply 0.

User I was trying to ssh into didn't have a default shell.

I ran the following:

chsh -s $(which sh) username 

And then I was able to ssh.

Note:

Running su username was returning exit-code 1 (was failing) and now it just works.

1
  • 1
    Holy crap that's cryptic! This was my problem as well. I tried making a "system" user (no HOME and no SHELL) to use SFTP with and I could authenticate but I couldn't use SFTP or SCP or SSH. This fixed my issue. Thanks!
    – Dave
    Commented Feb 3, 2017 at 19:14
4

In my case it was the zsh shell that stopped working on the server end. However, I was still able to send commands to verify it: ssh -t username@host 'echo $0'

Requested bash ssh -t username@host 'bash -l' stopped the server from terminating the session.

2
  • This worked to stop it from exiting. But still don't know why it exited. Is the fact that this worked insight into the why? Commented Mar 21, 2021 at 18:44
  • Might be your .zshrc or profile or another file that is automatically run after logging in that triggers your problem.
    – mateuszb
    Commented Mar 22, 2021 at 13:31
3

I was changed the Open files configuration in kernel parameter file /etc/security/limits.conf to unlimited and lost the connectivity.

After reverting it back to the normal for root user i got the connectivity back.

Wrong Example:
## Example hard limit for max opened files
*        hard   nofile unlimited
root     hard   nofile  unlimited
## Example soft limit for max opened files
*        soft   nofile unlimited
root     soft   nofile unlimited

Correct Ex:
## Example hard limit for max opened files
*        hard   nofile 16000
root     hard   nofile 16000
## Example soft limit for max opened files
*        soft   nofile 16000
root     soft   nofile 16000
2

I have encountered this on Mac OS X where the configuration in ~/.bashrc had an problem which caused ssh to work, but sftp to not work. @stéphane-chazelas seems to have the right idea in the comments above.

On the remote system via SSH, rename ~/.bashrc to ~/.bashrc-MOVED and try it again and see if it works; then restore ~/.bashrc and determine the issue.

On my system the ~/.bashrc contained this:

if [ -z "$PS1" ] ; then
    exit
fi

Which was the likely culprit.

1

I had the same issue today. First thing I noticed was /var/log was at 100%.

I fixed that and it didn't solve the issue. I couldn't ssh in nor could I login via the GUI, but I could Ctrl+Alt+F2 to get to CLI and login that way. I typed startx and received an error that /tmp/.X0-lock existed.

I removed that file (technically I removed everything from /tmp) and I was able to login via GUI and also via ssh.

1
  • Thanks! In my case it was /home filled up to 100% which I have detected by using df -h on CentOS 7.
    – RAM237
    Commented Aug 10, 2018 at 16:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .