Determine dynamically allocated port for OpenSSH RemoteForward

Question

Question (TL;DR)

When assigning ports dynamically for remote forwarding (a.k.a. -R option), how can a script on the remote machine (for instance sourced from .bashrc) determine which ports were chosen by OpenSSH?

---

Background

I use OpenSSH (on both ends) to connect to our central server, that I share with multiple other users. For my remote session (for now) I would like to forward X, cups and pulseaudio.

The most trivial is forwarding X, using the -X option. The allocated X address is stored in the environmental variable DISPLAY and from that I can determine the corresponding TCP port, in most cases anyways. But I hardly ever need to, because Xlib honors DISPLAY.

I need a similar mechanism for cups and pulseaudio. The basics for both services exist, in the form of the environmental variables CUPS_SERVER and PULSE_SERVER, respectively. Here are usage examples:

ssh -X -R12345:localhost:631 -R54321:localhost:4713 datserver

export CUPS_SERVER=localhost:12345
lowriter #and I can print using my local printer
lpr -P default -o Duplex=DuplexNoTumble minutes.pdf #printing through the tunnel
lpr -H localhost:631 -P default -o Duplex=DuplexNoTumble minutes.pdf #printing remotely

mpg123 mp3s/van_halen/jump.mp3 #annoy co-workers
PULSE_SERVER=localhost:54321 mpg123 mp3s/van_halen/jump.mp3 #listen to music through the tunnel

The problem is setting CUPS_SERVER and PULSE_SERVER correctly.

We use port forwardings a lot and therefore I need dynamic port allocations. Static port allocations are not an option.

OpenSSH has a mechanism for dynamic port allocation on the remote server, by specifying 0 as bind-port for remote forwarding (the -R option). By using the following command, OpenSSH will dynamically allocate ports for cups and pulse forwarding.

ssh -X -R0:localhost:631 -R0:localhost:4713 datserver

When I use that command, ssh will print the following to STDERR:

Allocated port 55710 for remote forward to 127.0.0.1:4713
Allocated port 41273 for remote forward to 127.0.0.1:631

There is the information I want! Ultimately I want to generate:

export CUPS_SERVER=localhost:41273
export PULSE_SERVER=localhost:55710

However the "Allocated port ..." messages are created on my local machine and sent to STDERR, which I can't access on the remote machine. Oddly enough OpenSSH does not seem to have means to retrieve Information about port forwardings.

How do I fetch that information to put it into a shell script to adequately set CUPS_SERVER and PULSE_SERVER on the remote host?

---

Dead Ends

The only easy thing I could find was increasing verbosity of the sshd until that information can be read from the logs. This is not viable as that information discloses a lot more information than is sensible to make accessible by non-root users.

I was thinking about patching OpenSSH to support an additional escape sequence which prints a nice representation of the internal struct permitted_opens, but even if that is what I want, I still can't script accessing the client escape sequences from the server side.

---

There must be a better way

The following approach seems very unstable and is limited to one such SSH session per user. However, I need at least two concurrent such sessions and other users even more. But I tried ...

When the stars are aligned properly, having sacrificed a chicken or two, I can abuse the fact that sshd is not started as my user, but drops privileges after successful login, to do this:

• get a list of port numbers for all listening sockets that belong to my user

  netstat -tlpen | grep ${UID} | sed -e 's/^.*:\([0-9]\+\) .*$/\1/'

• get a list of port numbers for all listening sockets that belong to processes my user started

  lsof -u ${UID} 2>/dev/null | grep LISTEN | sed -e 's/.*:\([0-9]\+\) (LISTEN).*$/\1/'

• All ports that are in the first set, but not in the second set have a high likelyhood to be my forwarding ports, and indeed subtracting the sets yields 41273, 55710 and 6010; cups, pulse and X, respectively.

• 6010 is identified as the X port using DISPLAY.

• 41273 is the cups port, because lpstat -h localhost:41273 -a returns 0.
• 55710 is the pulse port, because pactl -s localhost:55710 stat returns 0. (It even prints the hostname of my client!)

(To do the set substraction I sort -u and store the output from the above command lines and use comm to do the substraction.)

Pulseaudio lets me identify the client and, for all intents and purposes, this may serve as an anchor to separate SSH sessions that need separating. However, I haven't found a way to tie 41273, 55710 and 6010 to the same sshd process. netstat won't disclose that information to non-root users. I only get a - in the PID/Program name column where I would like to read 2339/54 (in this particular instance). So close ...

Answer 1

Unfortunately I didn't find your question earlier, but I've just got a really good answer from kamil-maciorowski:

https://unix.stackexchange.com/a/584505/251179

In summary, establish a master connection first, keep that in the background, then issue a second command with -O *ctl_cmd* set to forward to request/setup port forwarding:

ssh -fNMS /path/to/socket user@server

port="$(ssh -S /path/to/socket -O forward -R 0:localhost:22 placeholder)"

This will give you $port on your local machine, and a connection in the background.

You can then use $port locally; or use ssh again to run a command on the remote server, where you can use the same control socket.

As to the flags, these are:

• -f = Requests ssh to go to background.
• -N = Do not execute a remote command.
• -M = Places client into “master” mode, for connection sharing.
• -S = Location of a control socket for connection sharing.
• -O = Control an active connection multiplexing master process.

In my case I've added a bit more to keep checking the connection:

#!/bin/bash

#--------------------------------------------------
# Setup
#--------------------------------------------------

  set -u;

  tunnel_user="user";
  tunnel_host="1.1.1.1";

  local_port="22";
  local_name="my-name";

  path_key="$HOME/.ssh/tunnel_ed25519";

  path_lock="/tmp/tunnel.${tunnel_host}.pid"
  path_port="/tmp/tunnel.${tunnel_host}.port"
  path_log="/tmp/tunnel.${tunnel_host}.log"
  path_socket="/tmp/tunnel.${tunnel_host}.socket"

#--------------------------------------------------
# Key file
#--------------------------------------------------

  if [ ! -f "${path_key}" ]; then

    ssh-keygen -q -t ed25519 -f "${path_key}" -N "";

    /usr/local/bin/tunnel-client-key.sh
      # Sends the public key to a central server, also run via cron, so it can be added to ~/.ssh/authorized_keys
      # curl -s --form-string "pass=${pass}" --form-string "name=$(local_name)" -F "key=@${path_key}.pub" "https://example.com/key/";

  fi

#--------------------------------------------------
# Lock
#--------------------------------------------------

  if [ -e "${path_lock}" ]; then
    c=$(pgrep -F "${path_lock}" 2>/dev/null | wc -l);
      # MacOS 10.15.4 does not support "-c" to count processes, or the full "--pidfile" flag.
  else
    c=0;
  fi

  if [[ "${c}" -gt 0 ]]; then
    if tty -s; then
      echo "Already running";
    fi;
    exit;
  fi;

  echo "$$" > "${path_lock}";

#--------------------------------------------------
# Port forward
#--------------------------------------------------

  retry=0;

  while true; do

    #--------------------------------------------------
    # Log cleanup
    #--------------------------------------------------

      if [ ! -f "${path_log}" ]; then
        touch "${path_log}";
      fi

      tail -n 30 "${path_log}" > "${path_log}.tmp";

      mv "${path_log}.tmp" "${path_log}";

    #--------------------------------------------------
    # Exit old sockets
    #--------------------------------------------------

      if [ -S "${path_socket}" ]; then

        echo "$(date) : Exit" >> "${path_log}";

        ssh -S "${path_socket}" -O exit placeholder;

      fi

    #--------------------------------------------------
    # Lost lock
    #--------------------------------------------------

      if [ ! -e "${path_lock}" ] || ! grep -q "$$" "${path_lock}"; then

        echo "$(date) : Lost Lock" >> "${path_log}";

        exit;

      fi

    #--------------------------------------------------
    # Master connection
    #--------------------------------------------------

      echo "$(date) : Connect ${retry}" >> "${path_log}";

      ssh -o StrictHostKeyChecking=no -o ConnectTimeout=10 -o ServerAliveInterval=30 -o ExitOnForwardFailure=yes -fNTMS "${path_socket}" -i "${path_key}" "${tunnel_user}@${tunnel_host}" >> "${path_log}" 2>&1;

    #--------------------------------------------------
    # Setup and keep checking the port forwarding
    #--------------------------------------------------

      old_port=0;

      while ssh -S "${path_socket}" -O check placeholder 2>/dev/null; do

        new_port=$(ssh -S "${path_socket}" -O forward -R "0:localhost:${local_port}" placeholder 2>&1);

        if [[ "${new_port}" -gt 0 ]]; then

          retry=0;

          if [[ "${new_port}" -ne "${old_port}" ]]; then

            ssh -i "${path_key}" "${tunnel_user}@${tunnel_host}" "tunnel.port.sh '${new_port}' '${local_name}'" >> "${path_log}" 2>&1;
              # Tell remote server what the port is, and local_name.
              # Don't use socket, it used "-N"; if done, a lost connection keeps sshd running on the remote host, even with ClientAliveInterval/ClientAliveCountMax.

            echo "$(date) : ${new_port}" >> "${path_log}";

            echo "${new_port}" > "${path_port}";

            old_port="${new_port}";

            sleep 1;

          else

            sleep 300; # Looks good, check again in 5 minutes.

          fi

        else # Not a valid port number (0, empty string, number followed by an error message, etc?)

          ssh -S "${path_socket}" -O exit placeholder 2>/dev/null;

        fi

      done

    #--------------------------------------------------
    # Cleanup
    #--------------------------------------------------

      if [ ! -f "${path_port}" ]; then
        rm "${path_port}";
      fi

      echo "$(date) : Disconnected" >> "${path_log}";

    #--------------------------------------------------
    # Delay before next re-try
    #--------------------------------------------------

      retry=$((retry+1));

      if [[ $retry -gt 10 ]]; then
        sleep 180; # Too many connection failures, try again in 3 minutes
      else
        sleep 5;
      fi

  done

Answer 2

I have achieved the same by creating a pipe on the local client, then redirecting stderr to the pipe which is also redirected to the input of ssh. It doesn't require multiple ssh connections to presume a free known port which could fail. This way the logon banner and "Allocated port ### ..." text is redirected to the remote host.

I have a simple script on the host getsshport.sh which is run on the remote host which reads the redirected input and parses out the port. As long as this script doesn't end, the ssh remote forward stays open.

local side

mkfifo pipe
ssh -R "*:0:localhost:22" user@remotehost "~/getsshport.sh" 3>&1 1>&2 2>&3 < pipe | cat > pipe

3>&1 1>&2 2>&3 is a little trick to swap stderr and stdout, so that stderr gets piped to cat, and all the normal output from ssh is shown on stderr.

remote side ~/getsshport.sh

#!/bin/sh
echo "Connection from $SSH_CLIENT"
while read line
do
    echo "$line" # echos everything sent back to the client
    echo "$line" | sed -n "s/Allocated port \([0-9]*\) for remote forward to \(.*\)\:\([0-9]*\).*/client port \3 is on local port \1/p" >> /tmp/allocatedports
done

I did try to grep the "allocated port" message on the local side first before sending it through ssh, but seems that ssh will block waiting for the pipe to open on stdin. grep doesn't open pipe for writing until it receives something, so this basically deadlocks. cat however doesn't seem to have this same behaviour, and opens the pipe for writing immediately allowing ssh to open the connection.

this is the same problem on the remote side, and why read line by line instead of just grep from stdin - otherwise `/tmp/allocatedports' isn't written out until the ssh tunnel is closed which defeats the entire purpose

Piping the stderr of ssh into a command like ~/getsshport.sh is preferred, as without specifying a command, the banner text, or whatever else is in the pipe gets executed on the remote shell.

Answer 3

Take two (see history for a version which does scp from server side and is a bit simpler), this should do it. The gist of it is this:

1. pass an environment variable from client to server, telling server how it can detect when port information is avaiable and then get and use it.
2. once port information is available, copy it from client to server, allowing server to get it (with help of part 1 above), and use it

---

First, setup on remote side, you need to enable sending an env variable in sshd configuration:

sudo yourfavouriteeditor /etc/ssh/sshd_config

Find line with AcceptEnv and add MY_PORT_FILE to it (or add the line under right Host section if there isn't one yet). For me the line became this:

AcceptEnv LANG LC_* MY_PORT_FILE

Also remember to restart sshd for this to take effect.

Additionally, for below scripts to work, do mkdir ~/portfiles on remote side!

---

Then on local side, a script snippet which will

1. create temp file name for stderr redirection
2. leave a background job to wait for the file to have content
3. pass file name to server as env variable, while redirecting ssh stderr to the file
4. background job proceeds to copy the stderr temp file to server side using separate scp
5. background job also copies a flag file to server to indicate stderr file is ready

The script snippet:

REMOTE=$USER@datserver

PORTFILE=`mktemp /tmp/sshdataserverports-$(hostname)-XXXXX`
test -e $PORTFILE && rm -v $PORTFILE

# EMPTYFLAG servers both as empty flag file for remote side,
# and safeguard for background job termination on this side
EMPTYFLAG=$PORTFILE-empty
cp /dev/null $EMPTYFLAG

# this variable has the file name sent over ssh connection
export MY_PORT_FILE=$(basename $PORTFILE)

# background job loop to wait for the temp file to have data
( while [ -f $EMPTYFLAG -a \! -s $PORTFILE ] ; do
     sleep 1 # check once per sec
  done
  sleep 1 # make sure temp file gets the port data

  # first copy temp file, ...
  scp  $PORTFILE $REMOTE:portfiles/$MY_PORT_FILE

  # ...then copy flag file telling temp file contents are up to date
  scp  $EMPTYFLAG $REMOTE:portfiles/$MY_PORT_FILE.flag
) &

# actual ssh terminal connection    
ssh -X -o "SendEnv MY_PORT_FILE" -R0:localhost:631 -R0:localhost:4713 $REMOTE 2> $PORTFILE

# remove files after connection is over
rm -v $PORTFILE $EMPTYFLAG

---

Then a snippet for remote side, suitable for .bashrc:

# only do this if subdir has been created and env variable set
if [ -d ~/portfiles -a "$MY_PORT_FILE" ] ; then

       PORTFILE=~/portfiles/$(basename "$MY_PORT_FILE")
       FLAGFILE=$PORTFILE.flag
       # wait for FLAGFILE to get copied,
       # after which PORTFILE should be complete
       while [ \! -f "$FLAGFILE" ] ; do 
           echo "Waiting for $FLAGFILE..."
           sleep 1
       done

       # use quite exact regexps and head to make this robust
       export CUPS_SERVER=localhost:$(grep '^Allocated port [0-9]\+ .* localhost:631[[:space:]]*$' "$PORTFILE" | head -1 | cut -d" " -f3)
       export PULSE_SERVER=localhost:$(grep '^Allocated port [0-9]\+ .* localhost:4713[[:space:]]*$' "$PORTFILE" | head -1 | cut -d" " -f3)
       echo "Set CUPS_SERVER and PULSE_SERVER"

       # copied files served their purpose, and can be removed right away
       rm -v -- "$PORTFILE" "$FLAGFILE"
fi

---

Note: Above code is of course not very thoroughly tested and could contain all kinds of bugs, copy-paste errors, etc. Anybody using it better also understand it, use at your own risk! I tested it using just localhost connection, and it worked for me, in my test env. YMMV.

Answer 4

A snippet for local side, suitable for .bashrc:

#!/bin/bash

user=$1
host=$2

sshr() {
# 1. connect, get dynamic port, disconnect  
port=`echo "exit" | ssh -R '*:0:127.0.0.1:52698' -t $1 2>&1 | grep 'Allocated port' | awk '/port/ {print $3;}'`
# 2. reconnect with this port and set remote variable
cmds="ssh -R $port:127.0.0.1:52698 -t $1 bash -c \"export RMATE_PORT=$port; bash\""
($cmds)
}

sshr $user@$host

Answer 5

This is a tricky one, extra server-side handling along the lines of SSH_CONNECTION or DISPLAY would be great, but it's not easy to add: part of the problem is that only the ssh client knows the local destination, the request packet (to the server) contains only the remote address and port.

The other answers here have various unpretty solutions for capturing this client side and sending it to the server. Here's an alternate approach which isn't very much prettier to be honest, but at least this ugly party is kept on the client side ;-)

• client side, add/amend SendEnv so we can send some environment variables natively over ssh (probably not default)
• server side, add/amend AcceptEnv to accept same (probably not enabled by default)
• monitor the ssh client stderr output with a dynamically loaded library, and update the ssh client environment during the connection set up
• pick up the environment variables server side in the profile/login script

This works (happily, for now anyway) because the remote forwards are set up and recorded before the environment is exchanged (confirm with ssh -vv ...). The dynamically loaded library has to capture the write() libc function (ssh_confirm_remote_forward() → logit() → do_log() → write()). Redirecting or wrapping functions in an ELF binary (without recompiling) is orders of magnitude more complex than doing the same for a function in a dynamic library.

On the client .ssh/config (or command line -o SendEnv ...)

Host somehost
  user whatever
  SendEnv SSH_RFWD_*

On the server sshd_config (root/administrative change required)

AcceptEnv LC_* SSH_RFWD_*

This approach works for Linux clients and requires nothing special on the server, it should work for other *nix with some minor tweaks. Works from at least OpenSSH 5.8p1 up to 7.5p1.

Compile with gcc -Wall -shared -ldl -Wl,-soname,rfwd -o rfwd.so rfwd.c Invoke with:

LD_PRELOAD=./rfwd.so ssh -R0:127.0.0.1:4713 -R0:localhost:631 somehost

The code:

#define _GNU_SOURCE
#include <stdio.h>
#include <dlfcn.h>
#include <string.h>
#include <stdlib.h>

// gcc -Wall -shared  -ldl -Wl,-soname,rfwd -o rfwd.so rfwd.c

#define DEBUG 0
#define dfprintf(fmt, ...) \
    do { if (DEBUG) fprintf(stderr, "[%14s#%04d:%8s()] " fmt, \
          __FILE__, __LINE__, __func__,##__VA_ARGS__); } while (0)

typedef ssize_t write_fp(int fd, const void *buf, size_t count);
static write_fp *real_write;

void myinit(void) __attribute__((constructor));
void myinit(void)
{
    void *dl;
    dfprintf("It's alive!\n");
    if ((dl=dlopen(NULL,RTLD_NOW))) {
        real_write=dlsym(RTLD_NEXT,"write");
        if (!real_write) dfprintf("error: %s\n",dlerror());
        dfprintf("found %p write()\n", (void *)real_write);
    } else {
        dfprintf(stderr,"dlopen() failed\n");
    }
}

ssize_t write(int fd, const void *buf, size_t count)
{
     static int nenv=0;

     // debug1: Remote connections from 192.168.0.1:0 forwarded to local address 127.0.0.1:1000
     //  Allocated port 44284 for remote forward to 127.0.0.1:1000
     // debug1: All remote forwarding requests processed
     if ( (fd==2) && (!strncmp(buf,"Allocated port ",15)) ) {
         char envbuf1[256],envbuf2[256];
         unsigned int rport;
         char lspec[256];
         int rc;

         rc=sscanf(buf,"Allocated port %u for remote forward to %256s",
          &rport,lspec);

         if ( (rc==2) && (nenv<32) ) {
             snprintf(envbuf1,sizeof(envbuf1),"SSH_RFWD_%i",nenv++);
             snprintf(envbuf2,sizeof(envbuf2),"%u %s",rport,lspec);
             setenv(envbuf1,envbuf2,1);
             dfprintf("setenv(%s,%s,1)\n",envbuf1,envbuf2);
         }
     }
     return real_write(fd,buf,count);
}

(There are some glibc bear traps related to symbol versioning with this approach , but write() doesn't have this problem.)

If you're feeling brave, you could take the setenv() related code and patch it into ssh.c ssh_confirm_remote_forward() callback function.

This sets environment variables named SSH_RFWD_nnn, inspect these in your profile, e.g. in bash

for fwd in ${!SSH_RFWD_*}; do
    IFS=" :" read lport rip rport <<< ${!fwd}
    [[ $rport -eq "631" ]] && export CUPS_SERVER=localhost:$lport
    # ...
done

Caveats:

• there's not much error checking in the code
• changing the environment may cause thread-related problems, PAM uses threads, I don't expect problems but I haven't tested that
• ssh currently does not clearly log full forwarding of the form * local:port:remote:port* (if needed further parsing of debug1 messages with ssh -v would be required), but you don't need this for your use case

---

Oddly enough OpenSSH does not seem to have means to retrieve Information about port forwardings.

You can (partly) do this interactively with the escape ~#, oddly the implementation skips over channels that are listening, it only lists open (i.e. TCP ESTABLISHED) ones, and it doesn't print the useful fields in any case. See channels.c channel_open_message()

You can patch that function to print the details for SSH_CHANNEL_PORT_LISTENER slots, but that only gets you the local forwardings (channels are not the same thing as actual forwards). Or, you can patch it to dump the two forwarding tables from the global options struct:

#include "readconf.h"
Options options;  /* extern */
[...]
snprintf(buf, sizeof buf, "Local forwards:\r\n");
buffer_append(&buffer, buf, strlen(buf));
for (i = 0; i < options.num_local_forwards; i++) {
     snprintf(buf, sizeof buf, "  #%d listen %s:%d connect %s:%d\r\n",i,
       options.local_forwards[i].listen_host,
       options.local_forwards[i].listen_port,
       options.local_forwards[i].connect_host,
       options.local_forwards[i].connect_port);
     buffer_append(&buffer, buf, strlen(buf));
}
snprintf(buf, sizeof buf, "Remote forwards:\r\n");
buffer_append(&buffer, buf, strlen(buf));
for (i = 0; i < options.num_remote_forwards; i++) {
     snprintf(buf, sizeof buf, "  #%d listen %s:%d connect %s:%d\r\n",i,
       options.remote_forwards[i].listen_host,
       options.remote_forwards[i].listen_port,
       options.remote_forwards[i].connect_host,
       options.remote_forwards[i].connect_port);
     buffer_append(&buffer, buf, strlen(buf));
}

This works fine, though it's not a "programmatic" solution, with the caveat that the client code does not (yet, it's flagged XXX in the source) update the list when you add/remove forwardings on-the-fly (~C)

---

If the server(s) are Linux you have one more option, this is the one I use generally, though for local forwarding rather than remote. lo is 127.0.0.1/8, on Linux you can transparently bind to any address in 127/8, so you can used fixed ports if you use unique 127.x.y.z addresses, e.g.:

mr@local:~$ ssh -R127.53.50.55:44284:127.0.0.1:44284 remote
[...]
mr@remote:~$ ss -atnp src 127.53.50.55
State      Recv-Q Send-Q        Local Address:Port          Peer Address:Port 
LISTEN     0      128            127.53.50.55:44284                    *:*    

This is subject to binding privileged ports <1024, OpenSSH doesn't support Linux capabilities and has a hard-coded UID check on most platforms.

Wisely chosen octets (ASCII ordinal mnemonics in my case) help untangle the mess at the end of the day.