2

I'm running ssh running on macOS to redirect connections to local Unix domain socket to a domain socket on another machine. The command line for ssh call is roughly the following:

$ ssh -nNT -L /var/run/some.socket:/var/run/some.socket -o TCPKeepAlive=yes \
    -o ServerAliveCountMax=10 -o ServerAliveInterval=60 user@destination

After performing some load testing, I discovered that on occasion some client connections fail, and upon examining logs, I found the following error output from ssh at the same time that the connections fail:

channel 41: open failed: connect failed: open failed
channel 44: open failed: connect failed: open failed
channel 47: open failed: connect failed: open failed
channel 49: open failed: connect failed: open failed
channel 51: open failed: connect failed: open failed
channel 59: open failed: connect failed: open failed
channel 62: open failed: connect failed: open failed
channel 64: open failed: connect failed: open failed

The load test parameters are to run 100 concurrent connections (connect, send some data, receive some data, disconnect with total of 10,000 connections to be performed.)

The behavior observed is that at the beginning of the test when first set of connections are created very quickly, few connections fail with the above errors. How many fails ranges from run to run but usually between couple to a dozen or so. Most failures tend to happen at the beginning of the test, though at times occur later in the test (i.e. after first 100 had been made).

Other posts on SO with similar descriptions seem to be covering the issue of using localhost with workaround to use 127.0.0.1, which here makes it not relevant since it is not a TCP/IP socket. Also, the destination part in the above command is specified as an IP address already.

A bit at a loss on how to fix and trace the issue. I tried using -vvv to get detailed dumped of ssh operation with nothing fruitful (all it logs for the relevant channels is that socket was set to non-blocking).

Note that the call to ssh is done from a script, and the call is preceded with ulimit -n 1024 which should provide more than enough file descriptors to be available to service all the sockets.

Improve this question
2
  • Have you checked/set the file descriptors limit (for the ssh process)?
    – Ralph Rönnquist
    Jul 27, 2018 at 23:21
  • @RalphRönnquist, yes, call to ssh is proceeded in the script by a call ulimit -n 1024. Updating the question to put that clarification and that it is on macOS
    – LB2
    Jul 28, 2018 at 4:01

1 Answer 1

Reset to default
3 
channel 41: open failed: connect failed: open failed

This error message means that the remote SSH server was unable to perform a TCP forward request, because it couldn't connect to the target of the tunnel. The last "open failed" part of the message is an error message from the remote SSH server.

When you run SSH with a port forward, the port forward works like this:

  1. The local ssh client listens for TCP connections on the local port (/var/run/some.socket in your case).
  2. When an originator connects to the local port, the ssh client sends a request for a "direct-tcpip" channel to the server. The request includes the target of the tunnel (/var/run/some.socket on the remote system in your case).
  3. The remote SSH server makes a TCP connection to the target of the tunnel.
  4. The local ssh client and the remote ssh server relay data in both directions between the respective TCP connections and the direct-tcpip channel.

In your case, the ssh server is failing at step 3 because it can't connect to the target of the tunnel for some reason.

You should check the ssh log on the remote server. The SSH server process may have logged a message saying why it's failing. Aside from that, you say this is happening intermittently during a load test, so I'd look at server-side issues related to load. A couple of possibilities come to mind:

  1. The application on the remote system which is listening on /var/run/some.socket isn't handling connection requests quickly enough, and a backlog is building up.
  2. The SSH server process is hitting some kind of resource limit (number of open file descriptors, for example)
Improve this answer
6
  • I can see this message is produced in channel_input_open_failure(...) in channels.c file. This function is mapped as handler in both: clientloop.c and serverloop.c. But if it is generated on the server side (sshd), I don't see how it would make it to client (ssh) where it is logged. I can't find code that would do that. That function would be "called" either by server_input_channel_open or client_input_channel_open or channel_post_connecting, which seem to be functions for client- and server-sides. So if I see it from ssh, wouldn't it suggest that it is client-side (ssh) issue?
    – LB2
    Jul 30, 2018 at 15:44
  • Also, the logs show that the request for type of connection is [email protected], and not direct-tcpip
    – LB2
    Jul 30, 2018 at 16:46
  • SSH supports tunneling in both directions, client->server and server->client. So both the client and the server have logic to emit and handle messages about TCP forward channels. channel_input_open_failure() is just the function that formats and logs the error message which you're getting. It's being called from the client loop because the client received a message from the server saying it couldn't complete a request to open a TCP tunnel.
    – Kenster
    Jul 30, 2018 at 17:00
  • And [email protected] is an OpenSSH extension for tunnels involving unix domain sockets; the normal direct-tcpip channel type only supports the TCP domain.
    – Kenster
    Jul 30, 2018 at 17:01
  • So where's the code that sends error info from server side, and code on client side to receive said error info and pass it to the formatting function? I can't seem to find that functionality anywhere.
    – LB2
    Jul 30, 2018 at 19:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .