9

My goal is to allow 10000 concurrent sshs running on a single server.

For simplicity I am ssh-ing to localhost:

for i in `seq 1 10000`; do
    ssh localhost "echo ${i}; sleep 100"  >>./info 2>>./log &
done

sleep 100 is to make sure when the 10000th ssh starts, the 1st ssh is still in connection so there are indeed 10000 concurrent sshs.

And here are the two kinds of error messages I got:

1. ssh_exchange_identification: Connection closed by remote host

2. ssh_exchange_identification: read: Connection reset by peer

I have done the following modifications:

  1. In /etc/security/limits.conf and /etc/security/limits.d/90-nproc.conf, set soft&hard nofile&nproc to 65535 (this is the max possible value right? -- Update: no. The max value is 1048576)
  2. In /etc/sysctl.conf, set kernel.pty.max = 65535
  3. In /etc/ssh/sshd_config, set MaxStartups 10000.

These modifications allow me to successfully run 1000 concurrent sshs to a single server, but they don't work for 2000 and above sshs.

Some people have suggested changing the value for MaxSessions(actually I'm not clear about its usage: how does multiplexing affect my case?), /proc/sys/net/core/netdev_max_backlog and /proc/sys/net/core/somaxconn, but they seem to make no difference.

Besides, there's no error if they are 10000 concurrent sshs to different servers (problems occur only when ssh to a single server):

for i in `seq 1 10000`; do
    j=$(( 1 + $i % 8 ))
    ssh server-${j} "echo hi; sleep 100" >info-${j} 2>log-${j} &
done

I've been stuck on this for quite long.
Any help would be deeply appreciated!

  • 1
    sshd server log can provide more information about the reason for rejecting connections. Basically if you want just 10000 sessions, I would recommend you to use multiplexing using ControlMaster (and then of course bump MaxSessions). – Jakuje Aug 3 '15 at 11:17
  • 1
    I don't think the sleep 100s does what you think. It gets executed not in the ssh session, but on your own machine. – daniel kullmann Aug 3 '15 at 13:58
  • 1
    @Jakuje thanks for reminding me to check the server log! I found error: reexec socketpair: Too many open files, so I suppose the previous value of nofile (i.e. 65535) was far from enough. I am not familiar with ControlMaster but I'll try it, thank you!! :) – Clara Aug 4 '15 at 10:06
  • 1
    Interesting, when I do execute one of the lines, a ps axu | egrep "ssh|sleep" | grep -v grep only lists the sleep 100s, not the ssh. I think you should change the command to ssh "echo hi; sleep 100s". – daniel kullmann Aug 4 '15 at 13:29
  • 2
    @danielkullmann Yes you are absolutely right -- sleep 100 should be in the command sent through ssh, which is the case in my actual script, but I made a typo here. I have updated the main post accordingly. Thank you so much for pointing it out!! – Clara Aug 6 '15 at 2:09
2

/me wished he could comment

sshd needs to (typically, but though you didn't specifiy the exact use case(s) etc.) allocate a pty per login, however, in your case, ssh "echo hi; sleep 100s" does NOT allocate a pty, so no need for the kernel.pty.max setting... unless you want thousands of users loggedin*... to test that, you'll need to add the -t option to your tests, ie. ssh -t "echo hi; sleep 100s"

Back to the issue at hand with the error: reexec socketpair: Too many open files Tests on a Wheezy dist-upgraded to Jessie system, I found that /etc/security/limit* doesn't change the sshd's limits.

check that with cat /proc/<pid-of-sshd>/limits which in my case, after setting in /etc/security/limits.conf: * nofile soft 65535 * nofile hard 65535 still reports only 1024 (soft) and 4096(hard) for sshd's limits. The resolution appears to be to force the ulimit -Hn 65535 & ulimit -n 65535 inside the /etc/init.d/ssh script using that ulimit commands, I've raised the sshd's nofiles to 65535/65535 from 1024/4096

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.