4

so I have this problem, I can't login to my server with SSH. See:

ssh -v myuser@myserver
debug1: Authentication succeeded (password).
debug1: channel 0: new [client-session]
debug1: Requesting [email protected]
debug1: Entering interactive session.
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
Last login: Sun Jul  5 20:10:54 2015 from x.x.x.x
-bash: KSH_VERSION: unbound variable
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype [email protected] reply 0
debug1: channel 0: free: client-session, nchannels 1
Connection to x.x.x.x closed.
Transferred: sent 2264, received 2800 bytes, in 0.2 seconds
Bytes per second: sent 13241.8, received 16376.8
debug1: Exit status 1

The problem is the "-bash: KSH_VERSION: unbound variable", this kills the connection I think. I know why is it happening…I was working on a script placed in /etc/profile.d, and this script has "set -euo pipefail" (ouch) in it... The "o" option is "nounset" and that exaplains the "unbound variable"... When the script sets the options, it make the vim.sh there failing on the unbound variable.

Now, how get around this? I can't send just a command there over ssh, to remove the file for example, I can't login as another user, I can't use another shell, the "--noprofile" and "--norc" options for bash doesn't work…no trick I tried worked…

Any ideas, pretty please?

4
  • 1
    Does this work? ssh myuser@myserver hostname
    – Cyrus
    Commented Jul 5, 2015 at 20:29
  • nope, it doesn't :( same problem...
    – stibi
    Commented Jul 5, 2015 at 20:30
  • The man page for ssh (linux.die.net/man/1/ssh) documents a -N option to not execute a remote command. I don't know if there's a way to pass that to scp so you can update the file, but you might be able to forward ports to the remote machine and get access through them if for example, normal FTP is allowed from localhost. Commented Jul 5, 2015 at 22:46
  • If this is running Debian (or a derivative), you can try using /bin/dash as the shell instead. Not sure if it will work... Edit: Never mind, it reads /etc/profile. Commented Jul 6, 2015 at 23:09

7 Answers 7

5

I had the same problem and finally found a solution.

As @nkms mentioned:

You have about 1/3 chance (at least I get 1/3) to interrupt (hit Ctrl+C) the login process before /etc/profile is sourced

To raise you chances i've used the following loop:

$ (trap '' INT; while true; do ssh myuser@myserver; done)

The script starts subshell '(' and deactivates the ctr-c command for that subshell (with the trap command). Then it loops over the ssh command. If you continuously press ctrl-c or hold down ctrl-c, you raise you chances to hit the right moment.

... works like a charm!

2
  • 1
    It worked beautifully for me. Of all the things I've ever learned from this site, this one right here had the most "wow, my butt just got saved!" factor of any of them. Thank you for your clever solution. Commented Dec 30, 2018 at 23:06
  • @MarkThomson: THX!
    – st0ne
    Commented Jan 4, 2019 at 9:38
3

You have about 1/3 chance (at least I get 1/3) to interrupt (hit Ctrl+C) the login process before /etc/profile is sourced:

$ ssh -v localhost
OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013
........
Authenticated to localhost ([::1]:22).
debug1: channel 0: new [client-session]
debug1: Requesting [email protected]
debug1: Entering interactive session.
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
Last login: Tue Jul  7 19:46:42 2015 from localhost
^C-bash-4.2$ 

Edit to clarify a few things:

When sshd opens a session on the server, it does so by reading the user shell (that is /bin/bash in this case).

It does not matter whatever option is given on bash or ssh command to run (including sftp). That ssh command (including sftp) or bash option is passed to a bash instance after a session is opened. You have no control on what that instance does. For example:

client$ ssh localhost bash --noprofile --norc

strace output on the server:

[pid 60047] execve("/bin/bash", ["bash", "-c", "bash --noprofile --norc"], [/* 16 vars */]) = 0
[pid 60047] execve("/usr/bin/bash", ["bash", "--noprofile", "--norc"], [/* 28 vars */]) = 0

As you can see the ssh command ("bash --noprofile --norc") is passed on the first execve as an argument to "bash" with the "-c" option.

That first bash instance that you have no control on with read ~/.bashrc which in turn will read /etc/bashrc which will load the scripts under /etc/profile.d/ (at least on a Red Hat system - I don't know about others).

[pid 60047] open("/home/user/.bashrc", O_RDONLY) = 3
[pid 60047] open("/etc/bashrc", O_RDONLY) = 3

The only way I can think of is the above, been quick enough to abort the login process before the scripts are loaded (but not to quick as to abort the session). - You might want to increase the load on the system (if it's possible) to give you more time during the login process.

ps. if you find the answer useful please mark it

2
  • nice, this is cool
    – stibi
    Commented Jul 7, 2015 at 16:52
  • Tip: See @st0ne's supplemental contribution to this answer for a great way to increase your chances of Ctrl+C working. Commented Dec 30, 2018 at 23:12
0

Does sftp also fail ? If doesn't, you could make use of that to place a good /etc/profile.d file there.

5
  • good idea, but doesn't work either, it fail with "Connection closed"
    – stibi
    Commented Jul 5, 2015 at 20:37
  • Shucks. Is logging on via the console an option? Or use something other than ssh to logon (telnet, rsh, ftp by any chance?) Or if you've a backup solution like NetBackup, use that to restore a "good" version of the file?
    – steve
    Commented Jul 5, 2015 at 20:44
  • For a console, I'll have to ask the provider of the virtual machine. I'll try the other options, but doubt it...
    – stibi
    Commented Jul 5, 2015 at 20:46
  • 3
    What about scp? It works at a lower level than sftp. Something like scp file root@machine:/etc/profile.d. Commented Jul 5, 2015 at 20:47
  • scp seems to be also invoking bash process, because it fail in the same manner as all the other attempts…
    – stibi
    Commented Jul 5, 2015 at 22:15
0

I have two problems with your question.

ps -A|grep bash|wc -l

and the same command in

ssh localhost 'csh -l'

give the same number.

So it does not seem that a bash is run before 'csh -l'.

On the other hand csh does not use /etc/profile.

strace -eopen csh -l

does not show any entry of /etc/profile*.

So I am not sure the problem you have is in /etc/profile*.

Am I missing something?

4
  • 1
    the problem is definitelly there, I'm not 100% sure about the root cause, but I'll dig it out. Now I have the server ready and fixed again, provider's support helped me. Once the bloody script was gone, SSH worked again.
    – stibi
    Commented Jul 7, 2015 at 16:45
  • So just deleting a script in /etc/profile.d/ solved the problem? Commented Jul 7, 2015 at 16:47
  • Yes. Without the script and its "set -euo", the other scripts in profile.d were invoked without problems, even these with unbounded variables…nothing what would cause exiting these "wrong" scripts with non zero status code and thus taking the ssh connection down with them. That's my current exaplanation :)
    – stibi
    Commented Jul 7, 2015 at 16:51
  • Jodka: sshd runs your configured shell with -c 'args from client' which in your case is bash -c 'csh -l'. When bash is given a single command in -c, it doesn't run it as a forked child process like usual, instead bash exec's it, so the process that sshd created now is csh. Try ssh localhost 'csh -l;echo done' to see the bash. Commented Jan 9, 2019 at 5:05
0

This is unlikely to work, but you could try sending a value for KSH_VERSION when you make the ssh connection:

KSH_VERSION=x ssh -v -o 'SendEnv=KSH_VERSION' myuser@myserver

The idea is to set the environment variable to something, so that bash won't complain that it's unset. I say it's unlikely to work because the default SSH server configuration wouldn't permit you to set this particular variable. This would only work if someone had reason to make that setting more lenient on this particular server.

1
  • you are correct in all points :) doesn't work, because of the reasons you exaplained
    – stibi
    Commented Jul 7, 2015 at 16:44
0

I ended up with the same problem, but was installing something else and fixed it once I removed the below:

/etc/sysctl.conf
fs.file-max=100000

Then

/etc/security/limits.conf:
*               soft    nofile          1000000
*               hard    nofile          1000000

Lastly

/etc/pam.d/common-session:
session required pam_limits.so
-1

Try ssh -t username@hostname /bin/sh --noprofile.

4
  • Tried that, doesn't work, I think it is because there is a main bash process created which is parent to the one started from the ssh command
    – stibi
    Commented Jul 5, 2015 at 20:52
  • 1
    And a different shell? Commented Jul 6, 2015 at 16:03
  • ksh, csh…both invoking /etc/profile, no go
    – stibi
    Commented Jul 7, 2015 at 6:49
  • What does csh -f? Commented Jul 7, 2015 at 9:49

Not the answer you're looking for? Browse other questions tagged .