12

The most recent thing I remember is changing the soft and hard memlock ulimit to unlimited. Now I can't ssh into the machine.

This is the ssh log.

Authenticated to IP ([IP]:22).
debug1: channel 0: new [client-session]
debug2: channel 0: send open
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug2: callback start
debug2: fd 3 setting TCP_NODELAY
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug1: Sending environment.
debug1: Sending env LC_CTYPE = 
debug2: channel 0: request env confirm 0
debug2: channel 0: request shell confirm 1
debug2: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel 0: rcvd adjust 2097152
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0
Last login: Wed Aug  6 07:18:07 2014 from IP-SOURCE
debug2: channel 0: rcvd eof
debug2: channel 0: output open -> drain
debug2: channel 0: obuf empty
debug2: channel 0: close_write
debug2: channel 0: output drain -> closed
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
debug2: channel 0: rcvd eow
debug2: channel 0: close_read
debug2: channel 0: input open -> closed
debug2: channel 0: rcvd close
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send close
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
Connection to IP closed.
Transferred: sent 4256, received 2504 bytes, in 0.4 seconds
Bytes per second: sent 9616.9, received 5658.0
debug1: Exit status 254

I have tried the following unsuccessfully till now, before posting here:

  1. Trying for a norc noprofile login by ssh user@host 'bash --noprofile'

  2. Forcing a tty by ssh -t user@host

  3. Moved the bash_profile. Tried sshing by ssh user@host.

  4. Renaming the limits.conf file in hopes that it won't be read.

  5. Restarted ssh server.

  6. Run a command via knife as knife ssh "name:server" "come_command"

  7. ssh user@host 'ulimit -l 64', ssh user@host 'ulimit -S -l 64', ssh user@host 'ulimit -H -l 64', ssh user@host 'exec ulimit -H -l 64'

I am not sure if this way of running commands inline: ssh user@host "some_command" works, because I can't get a simple directory listing. I also tried rebooting by ssh user@host 'reboot' but don't think the command was executed. I restarted the machine from AWS also, but unsuccessful.

Is it a lost cause trying to ssh? Is there any way I can ssh into the server?

  • 1
    If the login shell of the remote user is bash, then bash always reads ~/.bashrc over ssh. There's no way around that. Can you sftp there and check/alter your bashrc that way? – Stéphane Chazelas Aug 6 '14 at 12:07
  • Tried sftp using CyberDuck, got the SSH_FXP_INIT error code. – theTuxRacer Aug 6 '14 at 12:09
  • You can also get more detailed output using the -v option, or for more, the -vv option of for even more, then -vvv option. E.g. ssh -vvv user@host. That may give you a better idea of where things are going wrong. – Warwick Aug 6 '14 at 12:43
  • Tried that. This is where I got the log from. – theTuxRacer Aug 6 '14 at 13:13
  • Do you have any other file (FTP/HTTP...?) or shell (console?) access to the machine which you can use? – Stéphane Chazelas Aug 6 '14 at 15:14
12

Try to change

UsePAM yes

on

UsePAM no

in /etc/ssh/sshd_config (for CentOS)

  • It works, but why? – FelikZ Jul 7 '15 at 11:12
  • 1
    sorry, but i dont remember the reason) – frad sorvensen Jul 8 '15 at 11:53
  • Is SELinux in play here ? Wondering if the context of /etc/security/limits.conf got hosed, and pam can't use it any more. – steve Jul 15 '16 at 19:46
  • If you are using a distribution using systemd this is a bad solution IMHO. This will prevent logindto a open a session and when/if the user is rebooting the machine, some processes started as the user will not be stopped as expected. – Bigon Dec 19 '16 at 19:30
  • In my case, the reason is the user Hard Open Fils Limit per processor is bigger than the nr_open. (nr_open is reset when restarting the machine): you can check the nr_open by cat /proc/sys/fs/nr_open, and hard open files ulimit -Hn. If you still want the ssh login user use the Hard Open File Config, you need to increase nr_open: sudo sysctl -w fs.nr_open=NUM_BIGGER_THAN_HARD – Xin Meng Mar 22 '19 at 14:14
4

I had a similar problem, I seemed to only see following strange message:

client_input_channel_req: channel 0 rtype exit-status reply 0.

User I was trying to ssh into didn't have a default shell.

I ran the following:

chsh -s $(which sh) username 

And then I was able to ssh.

Note:
Running su username was returning exit-code 1 (was failing) and now it just works.

  • 1
    Holy crap that's cryptic! This was my problem as well. I tried making a "system" user (no HOME and no SHELL) to use SFTP with and I could authenticate but I couldn't use SFTP or SCP or SSH. This fixed my issue. Thanks! – Dave Feb 3 '17 at 19:14
2

I have encountered this on Mac OS X where the configuration in ~/.bashrc had an problem which caused ssh to work, but sftp to not work. @stéphane-chazelas seems to have the right idea in the comments above.

On the remote system via SSH, rename ~/.bashrc to ~/.bashrc-MOVED and try it again and see if it works; then restore ~/.bashrc and determine the issue.

On my system the ~/.bashrc contained this:

if [ -z "$PS1" ] ; then
    exit
fi

Which was the likely culprit.

1

I had the same issue today. First thing I noticed was /var/log was at 100% I fixed that and it didn't solve the issue. I couldn't ssh in nor could I login via the GUI, but I could CNTRL+ALT+F2 to get to CLI and login that way. I typed startx and received an error that /tmp/.X0-lock existed.

I removed that file (technically I removed everything from /tmp) and I was able to login via GUI and also via ssh.

  • Thanks! In my case it was /home filled up to 100% which I have detected by using df -h on CentOS 7. – RAM237 Aug 10 '18 at 16:27
1

I was changed the Open files configuration in kernel parameter file /etc/security/limits.conf to unlimited and lost the connectivity.

After reverting it back to the normal for root user i got the connectivity back.

Wrong Example:
## Example hard limit for max opened files
*        hard   nofile unlimited
root     hard   nofile  unlimited
## Example soft limit for max opened files
*        soft   nofile unlimited
root     soft   nofile unlimited

Correct Ex:
## Example hard limit for max opened files
*        hard   nofile 16000
root     hard   nofile 16000
## Example soft limit for max opened files
*        soft   nofile 16000
root     soft   nofile 16000

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.